[ { "path": "LICENSE.txt", "content": "Copyright (c) 2017-2018\n Andreia Correia \n Pedro Ramalhete \n Pascal Felber \n Nachshon Cohen \n\nPermission is hereby granted, free of charge, to any person obtaining\na copy of this software and associated documentation files (the\n\"Software\"), to deal in the Software without restriction, including\nwithout limitation the rights to use, copy, modify, merge, publish,\ndistribute, sublicense, and/or sell copies of the Software, and to\npermit persons to whom the Software is furnished to do so, subject to\nthe following conditions:\n\nThe above copyright notice and this permission notice shall be\nincluded in all copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND,\nEXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF\nMERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND\nNONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE\nLIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION\nOF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION\nWITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\n" }, { "path": "README.md", "content": "\n# OneFile PTM / STM\n\nOneFile is a Software Transactional Memory (STM) meant to make it easy to implement lock-free and wait-free data structures.\nIt is based on the paper \"[OneFile: A Wait-free Persistent Transactional Memory](https://github.com/pramalhe/OneFile/blob/master/OneFile-2019.pdf)\" by Ramalhete, Correia, Felber and Cohen\nhttps://github.com/pramalhe/OneFile/blob/master/OneFile-2019.pdf\n\nIt provides multi-word atomic updates on *tmtype* objects, where T must be word-size, typically a pointer or integer.\nDuring a transaction, each store on an *tmtpye* is transformed into a double-word-compare-and-swap DCAS() and one more regular CAS() is done to complete the transaction. It does this with a store-log (write-set) which other writers can help apply. \nThis is a \"redo-log\" based technique, which means that both store and loads need to be interposed. Stores will be interposed to save them in the log and loads will be interposed to lookup on the log the most recent value.\nIf there is a transaction currently ongoing, the readers will have to check on each *tmtype::pload()* if the variable we're tying to read is part of the current transaction. If a value is read whose 'seq' is higher than the transaction we initially read, the whole read-only operation will be restarted, by throwing an exception in the *tmtype::pload()* interposing method and catching this exception by the TM. All of this logic is handled internally by OF without any explicit user interaction.\n\nBecause of operator overloading, the assignment and reading of *tmtype* types is done transparently to the user, with a pure library implementation, without any need for compiler instrumentation. This means that the user can write the code as if it was a sequential implementation of the data structure, apart from the change of types (type annotation). In this sense, OneFile is a \"quasi-universal construction\" with lock-free progress.\n\nOur design goal with OneFile was to provide a non-blocking STM so that non-experts could implement their own lock-free and wait-free data structures.\nOneFile is not designed to transform regular everyday code into lock-free applications. Such use-cases require a lot more of engineering work and likely a completely different approach from what we took with OneFile (CX is a much better option for that purpose).\n\nWe've made two implementations in the form of Persistent Transactional Memory (PTM) which are STMs meant for Persistent Memory, like Intel's Optane DC Persistent Memory.\n\nWe've implemented four diferent variants of this design:\n- OneFile-LF: The simplest of the four, has lock-free progress and lock-free memory reclamation using Hazard Eras;\n- OneFile-WF: Uses aggregation (like flat-combining) and a new wait-free consensus to provide wait-free bounded progress. Has wait-free bounded memory reclamation;\n- POneFile-LF: A PTM with durable transactions (ACID) and lock-free progress. Memory reclamation is lock-free using an optimistic technique. Allocation and de-allocation of user objects is lock-free;\n- POnefile-WF: A PTM with durable transactions (ACID) and wait-free progress. Memory reclamation for user objects is wait-free using an optimistic technique, while memory reclamation of the transactional objects is done using Hazard Eras, also wait-free. Allocation and de-allocation of user-objects is wait-free;\n\nSee the respective .hpp files for implementation details.\n\nEach implementation is a single header file. Yes, it's that small :)\n\n\n## Quickstart ##\n\nIf you just want to use OneFile in your own application or benchmarks then follow these steps:\n- Choose one of the four OneFile implementations, depending on whether you want and STM, a PTM, lock-free or wait-free progress: \n [stms/OneFileLF.hpp](https://github.com/pramalhe/OneFile/blob/master/stms/OneFileLF.hpp) STM with lock-free transactions \n [stms/OneFileWF.hpp](https://github.com/pramalhe/OneFile/blob/master/stms/OneFileWF.hpp) STM with wait-free transactions \n [ptms/POneFileLF.hpp](https://github.com/pramalhe/OneFile/blob/master/ptms/OneFilePTMLF.hpp) PTM with lock-free transactions \n [ptms/POneFileWF.hpp](https://github.com/pramalhe/OneFile/blob/master/ptms/OneFilePTMWF.hpp) PTM with wait-free transactions \n- Copy the header to your development folder\n- Include the header from a single .cpp. If you include from multiple compilation units (.cpp files) then move the last block in the .hpp to one of the .cpp files.\n- If you want a data structure that is already made then take a look at what's on these folders:\n\n\tdatastructures/ Data structures for volatile memory (needs one of the STMs)\n\tpdatastructures/ Data structures for persistent memory (needs one of the PTMs)\n\n\n### Design ###\n\nIn OneFile STM a transaction goes through three phases. \nThe first phase is to convert the operation (lambda) into a store-log (write-set). There is no need to save the loads (read-set) because unlike other approaches, a transaction does not need to re-check for changes at commit time: it does a check in-flight on each load of whether or not the value has changed since the beginning of the transaction, by looking at a sequence number associated with every read value, a technique similar to TL2 or TinySTM but without the need for keeping a read-set because all write transactions are executed one at a time, effectively serialized.\nThe second phase is to commit the transaction by advancing the current transaction (curTx).\nThe third phase is to apply the store-log using DCAS.\n\nThe first phase is implicitly serializable. Even if each thread publishes its operation, there is no way to parallelize this work among threads. The best that could be done would be that each thread to transform its own operation into its own store-log which it then appends to a global store-log. Unfortunately this is possible only for disjoint-access parallel transactions, and these are not easy to detect, therefore, our implementation of OneFile does not do this.\nInstead, we attempt to parallelize the second stage, where the store-log is applied. This task is easier to split among multiple threads, thus parallelizing it.\nAdding Flat-Combining or other similar aggregation techniques to the first stage, means that each thread will produce a store-log containing the operations of all other threads. This can be a bottleneck if the operations involves heavy computations and small store-logs. For data structures, this is not the case, and OneFile is designed to implement and work with data structures or other scenarios where the transactions are short in time, therefore, we found it acceptable to go with such an approach.\n\nThe parallelization of the third phase can be done with at least two different approaches: blocking and non-blocking.\nIn the blocking approach, the store-log can be divided into chunks (for example, one chunk per thread), each chunk having a lock, and the thread that takes the lock is responsible for applying that chunk.\nIn the non-blocking approach (OF-LF and OF-WF), each thread tries to apply an entry of the store-log at a time. To avoid ABA issues, a double-word compare-and-swap (DCAS) must be used.\n\nIn summary, OneFile does *not* do disjoint-access parallel transactions. If you absolutely need that functionality, then go and take a look at TinySTM. \n\n\n\n## Requirements ##\n\n- OneFile needs a double-word CAS, which limits it to x86. The algorithm can be modified to use LL/SC or even single-word CAS at the cost of losing its generic capability because bits would have to be stolen from a 64 bit wordl\n- The user must \"instrument\" the code where the atomic updates take place by wrapping the types with *tmtype*. Even then, the operator overloading will not cover all the cases and there will be situations where the user has to annotate the code with .pload() or .pstore() respectively;\n- The *T* type must be the size of a word, i.e. 64 bits. Anything bigger and it needs to be splitted into multiple *tmtype* objects;\n- If memory reclamation is needed, then the objects need to derive from the *tmbase* base class, need to be allocated with *tmNew()* or *tmMalloc()* and deallocated with *tmDelete()* or *tmFree()*;\n\n\n\n## Memory Reclamation ##\n\nWe're using a customized implementation of Hazard Eras, lock-free/wait-free memory reclamation: \n[https://github.com/pramalhe/ConcurrencyFreaks/blob/master/papers/hazarderas-2017.pdf](https://github.com/pramalhe/ConcurrencyFreaks/blob/master/papers/hazarderas-2017.pdf) \n[https://dl.acm.org/citation.cfm?id=3087588](https://dl.acm.org/citation.cfm?id=3087588) \nSee the HazardErasOF class in each implementation for more details. \n\nAs far as we know, there is only one wait-free data structure that has integrated wait-free memory reclamation: \n[https://github.com/pramalhe/ConcurrencyFreaks/blob/master/papers/crturnqueue-2016.pdf]([https://github.com/pramalhe/ConcurrencyFreaks/blob/master/papers/crturnqueue-2016.pdf) \nOneFile and CX are the first time that someone has made a generic mechanism for wait-free memory reclamation.\n\n\n\n## How to use this ##\n\n1. Annotate all the objects that are shared among threads, namely, everything that is *std::atomic* should be changed to *tmtype*;\n2. Use only *pstore()* and *pload()* (or just use '='). Do *not* call compare_exchange_strong(), exchange() or fetch_add();\n3. Replace calls to \"obj = new T(args)\" with \"obj = tmNew(args)\";\n4. Replace calls to \"delete obj\" with \"tmDelete(obj)\";\n5. The T types must derive from the base class *tmbase*;\n6. Place your methods in a lambda, capturing whatever you need, and pass the lambda to *updateTx()*;\nThat's it, you've now got your own lock-free data structure!\nFor an example of a simple linked-list set, take a look datastructures/linkedlists/OFLFLinkedListSet.hpp\n\n\n## Disadvantages ##\n\n- All mutative operations are serialized;\n- Types must be broken down to 64 bit sizes;\n- Requires Double-word-compare-and-swap (DCAS);\n\n\n## Advantages ##\n\n- Lock-free programming was never so easy, all the user code has to do is loads and stores on *tmtypes* types and those get transformed into a DCAS() based transaction that provides correct linearizable lock-free progress, without ABA issues;\n- Memory reclamation is also handled by OF using Hazard Eras, a lock-free/wait-free memory reclamation technique;\n- Compared to hand-written lock-free data structures, on the uncontended case, we are replacing each CAS with a DCAS and adding one extra (regular) CAS on the currTrans, which is a small price to pay for the atomicity;\n- This technique provides full linearizability for generic code, even mutative iterators, something which is nearly impossible to do with hand-written lock-free data structures;\n- Multiple helping threads can help apply starting on different places. A good heuristic is to start from the (tid % numStores);\n- OneFile-WF is the first STM with wait-free bounded progress and it's the first to have wait-free bounded progress with wait-free bounded memory reclamation.\n- Read-only transactions are lightweight and they can run concurrently with write transactions as long as they're disjoint;\n\nThe biggest advantage of all is that it's way easier to use OneFile than it is to implement a hand-made lock-free or wait-free data structure.\n\n\n\n## Examples ##\n\nThere are some working examples in the \"datastructures/\" folder:\nOFLFBoundedQueue.hpp: An array based queue (memory-bounded) \nOFLFLinkedListQueue.hpp: A linked list based queue (memory unbounded) \nOFLFLinkedListSet.hpp: A linked list based set \nOFLFRedBlackBST.hpp: A Red-Black (balanced) tree map \n\n\n\n## Benchmarks ##\nTo build the benchmarks you need to build ESTM and TinySTM, and then you need to pull PMDK (PMEM/NVML) and build it:\n\n cd ~/onefile/stms/\n cd estm-0.3.0\n make clean ; make\n cd ..\n cd tinystm\n make clean ; make\n cd ..\n cd ~\n git clone https://github.com/pmem/pmdk.git\n cd pmdk\n make -j12\n sudo make install\n export PMEM_IS_PMEM_FORCE=1\n cd ~/onefile/graphs\n make -j12\n \n\n## Tests ##\nThe four implementations of OneFile were executed during thousands of cpu hours and heavily stress tested with invariant checking and using tools like address sanitizer and valgrind. This is a lot more than what other STMs on github provide, but it doesn't mean there are no bugs in it ;)\nIf you see a crash or invariant failure, run the same code under a global rw-lock to make sure the bug is not in your code. If you really believe it's in OneFile, then please open a bug on github and add as much information as you can, namely, stack trace and files needed to reproduce. \nWe'll do our best to address it.\n\n \n" }, { "path": "common/HazardEras.hpp", "content": "/******************************************************************************\r\n * Copyright (c) 2016-2017, Pedro Ramalhete, Andreia Correia\r\n * All rights reserved.\r\n *\r\n * Redistribution and use in source and binary forms, with or without\r\n * modification, are permitted provided that the following conditions are met:\r\n * * Redistributions of source code must retain the above copyright\r\n * notice, this list of conditions and the following disclaimer.\r\n * * Redistributions in binary form must reproduce the above copyright\r\n * notice, this list of conditions and the following disclaimer in the\r\n * documentation and/or other materials provided with the distribution.\r\n * * Neither the name of Concurrency Freaks nor the\r\n * names of its contributors may be used to endorse or promote products\r\n * derived from this software without specific prior written permission.\r\n\r\n *\r\n * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\r\n * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED\r\n * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\r\n * DISCLAIMED. IN NO EVENT SHALL BE LIABLE FOR ANY\r\n * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\r\n * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\r\n * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND\r\n * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\r\n * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\r\n * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\r\n ******************************************************************************\r\n */\r\n\r\n#ifndef _HAZARD_ERAS_H_\r\n#define _HAZARD_ERAS_H_\r\n\r\n#include \r\n#include \r\n#include \r\n#include \r\n\r\n/*\r\n *

Hazard Eras

\r\n * This a light-weight implementation of hazard eras, where each thread has a\r\n * thread-local list of retired objects.\r\n *\r\n * This is based on the paper \"Hazard Eras - Non-Blocking Memory Reclamation\"\r\n * by Pedro Ramalhete and Andreia Correia:\r\n * github...\r\n *\r\n * The type T is for the objects/nodes and it's expected to have the following members:\r\n * newEra, delEra, delNext.\r\n *\r\n * R is zero.\r\n *\r\n *

\r\n * @author Pedro Ramalhete\r\n * @author Andreia Correia\r\n */\r\ntemplate\r\nclass HazardEras {\r\n\r\nprivate:\r\n static const uint64_t NONE = 0;\r\n static const int HE_MAX_THREADS = 128;\r\n static const int MAX_HES = 5; // This is named 'K' in the HP paper\r\n static const int CLPAD = 128/sizeof(std::atomic);\r\n static const int HE_THRESHOLD_R = 0; // This is named 'R' in the HP paper\r\n\r\n const int maxHEs;\r\n const int maxThreads;\r\n\r\n alignas(128) std::atomic eraClock {1};\r\n alignas(128) std::atomic* he[HE_MAX_THREADS];\r\n alignas(128) std::vector retiredList[HE_MAX_THREADS*CLPAD]; // It's not nice that we have a lot of empty vectors\r\n\r\npublic:\r\n HazardEras(int maxHEs=MAX_HES, int maxThreads=HE_MAX_THREADS) : maxHEs{maxHEs}, maxThreads{maxThreads} {\r\n for (int it = 0; it < HE_MAX_THREADS; it++) {\r\n he[it] = new std::atomic[CLPAD*2]; // We allocate four cache lines to allow for many hps and without false sharing\r\n retiredList[it*CLPAD].reserve(maxThreads*maxHEs);\r\n for (int ihe = 0; ihe < MAX_HES; ihe++) {\r\n he[it][ihe].store(NONE, std::memory_order_relaxed);\r\n }\r\n }\r\n static_assert(std::is_same::value, \"T::newEra must be uint64_t\");\r\n static_assert(std::is_same::value, \"T::delEra must be uint64_t\");\r\n }\r\n\r\n ~HazardEras() {\r\n for (int it = 0; it < HE_MAX_THREADS; it++) {\r\n delete[] he[it];\r\n // Clear the current retired nodes\r\n for (unsigned iret = 0; iret < retiredList[it*CLPAD].size(); iret++) {\r\n delete retiredList[it*CLPAD][iret];\r\n }\r\n }\r\n }\r\n\r\n\r\n inline uint64_t getEra() {\r\n return eraClock.load();\r\n }\r\n\r\n\r\n /**\r\n * Progress Condition: wait-free bounded (by maxHEs)\r\n */\r\n inline void clear(const int tid) {\r\n for (int ihe = 0; ihe < maxHEs; ihe++) {\r\n he[tid][ihe].store(NONE, std::memory_order_release);\r\n }\r\n }\r\n\r\n\r\n /**\r\n * Progress Condition: lock-free\r\n */\r\n inline T* get_protected(int index, const std::atomic& atom, const int tid) {\r\n auto prevEra = he[tid][index].load(std::memory_order_relaxed);\r\n\t\twhile (true) {\r\n\t\t T* ptr = atom.load();\r\n\t\t auto era = eraClock.load(std::memory_order_acquire);\r\n\t\t if (era == prevEra) return ptr;\r\n he[tid][index].store(era);\r\n prevEra = era;\r\n\t\t}\r\n }\r\n\r\n inline void protectEraRelease(int index, int other, const int tid) {\r\n auto era = he[tid][other].load(std::memory_order_relaxed);\r\n if (he[tid][index].load(std::memory_order_relaxed) == era) return;\r\n he[tid][index].store(era, std::memory_order_release);\r\n }\r\n\r\n\r\n /*\r\n * Does a single iteration. Must be integrated into the algorithm that's using HE.\r\n * In other words, we must re-check if era has changed\r\n *\r\n * Progress Condition: wait-free population oblivious\r\n */\r\n inline T* protectPtr(int index, const std::atomic& atom, uint64_t& prevEra, const int tid) {\r\n T* ptr = atom.load(std::memory_order_acquire);\r\n auto era = eraClock.load();\r\n if (prevEra != era) {\r\n prevEra = era;\r\n he[tid][index].store(era, std::memory_order_relaxed);\r\n std::atomic_thread_fence(std::memory_order_seq_cst);\r\n }\r\n return ptr;\r\n }\r\n\r\n\r\n /**\r\n * Retire an object (node)\r\n * Progress Condition: wait-free bounded\r\n */\r\n void retire(T* ptr, const int mytid) {\r\n auto currEra = eraClock.load();\r\n ptr->delEra = currEra;\r\n auto& rlist = retiredList[mytid*CLPAD];\r\n rlist.push_back(ptr);\r\n if (eraClock == currEra) eraClock.fetch_add(1);\r\n for (unsigned iret = 0; iret < rlist.size();) {\r\n auto obj = rlist[iret];\r\n if (canDelete(obj, mytid)) {\r\n rlist.erase(rlist.begin() + iret);\r\n delete obj;\r\n continue;\r\n }\r\n iret++;\r\n }\r\n }\r\n\r\nprivate:\r\n bool canDelete(T* obj, const int mytid) {\r\n for (int tid = 0; tid < maxThreads; tid++) {\r\n for (int ihe = 0; ihe < maxHEs; ihe++) {\r\n const auto era = he[tid][ihe].load(std::memory_order_acquire);\r\n if (era == NONE || era < obj->newEra || era > obj->delEra) continue;\r\n return false;\r\n }\r\n }\r\n return true;\r\n }\r\n\r\n};\r\n\r\n#endif /* _HAZARD_ERAS_H_ */\r\n" }, { "path": "common/HazardPointers.hpp", "content": "/******************************************************************************\r\n * Copyright (c) 2014-2016, Pedro Ramalhete, Andreia Correia\r\n * All rights reserved.\r\n *\r\n * Redistribution and use in source and binary forms, with or without\r\n * modification, are permitted provided that the following conditions are met:\r\n * * Redistributions of source code must retain the above copyright\r\n * notice, this list of conditions and the following disclaimer.\r\n * * Redistributions in binary form must reproduce the above copyright\r\n * notice, this list of conditions and the following disclaimer in the\r\n * documentation and/or other materials provided with the distribution.\r\n * * Neither the name of Concurrency Freaks nor the\r\n * names of its contributors may be used to endorse or promote products\r\n * derived from this software without specific prior written permission.\r\n\r\n *\r\n * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\r\n * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED\r\n * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\r\n * DISCLAIMED. IN NO EVENT SHALL BE LIABLE FOR ANY\r\n * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\r\n * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\r\n * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND\r\n * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\r\n * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\r\n * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\r\n ******************************************************************************\r\n */\r\n\r\n#ifndef _HAZARD_POINTERS_H_\r\n#define _HAZARD_POINTERS_H_\r\n\r\n#include \r\n#include \r\n\r\n\r\n\r\n/**\r\n * This is a customized version of Hazard Pointers to be used with CXMutation\r\n */\r\n// TODO: use std::vector instead of arrays for the retired objects (keep the padding)\r\ntemplate\r\nclass HazardPointers {\r\n\r\nprivate:\r\n static const int MAX_THREADS = 128;\r\n static const int MAX_HPS = 5;\r\n static const int MAX_RETIRED = MAX_THREADS*MAX_HPS;\r\n static const int HP_THRESHOLD_R = 0; // This is named 'R' in the HP paper\r\n static const int CLPAD = 128/sizeof(std::atomic);\r\n const int maxHPs;\r\n const int maxThreads;\r\n alignas(128) std::atomic* hp[MAX_THREADS*CLPAD];\r\n alignas(128) T** retiredObjects[MAX_THREADS*CLPAD]; // List of retired nodes that need to be 'deleted' for the current thread\r\n alignas(128) long numRetiredObjects[MAX_THREADS*CLPAD]; // Number of nodes in the retired list\r\n // Used specifically for CXMutation\r\n alignas(128) std::atomic heads[2*MAX_THREADS*CLPAD];\r\n\r\npublic:\r\n HazardPointers(int maxHPs=MAX_HPS, int maxThreads=MAX_THREADS) : maxHPs{maxHPs}, maxThreads{maxThreads} {\r\n for (int ih = 0; ih < 2*MAX_THREADS; ih++) {\r\n heads[ih*CLPAD].store(nullptr, std::memory_order_relaxed);\r\n }\r\n for (int ithread = 0; ithread < MAX_THREADS; ithread++) {\r\n \tnumRetiredObjects[ithread*CLPAD] = 0;\r\n hp[ithread*CLPAD] = new std::atomic[MAX_HPS];\r\n for (int ihp = 0; ihp < MAX_HPS; ihp++) {\r\n hp[ithread*CLPAD][ihp].store(nullptr, std::memory_order_relaxed);\r\n }\r\n retiredObjects[ithread*CLPAD] = new T*[MAX_RETIRED];\r\n for (int iret = 0; iret < MAX_RETIRED; iret++) {\r\n retiredObjects[ithread*CLPAD][iret] = nullptr;\r\n }\r\n }\r\n }\r\n\r\n ~HazardPointers() {\r\n for (int ithread = 0; ithread < MAX_THREADS; ithread++) {\r\n // Clear the current retired nodes\r\n for (int iret = 0; iret < numRetiredObjects[ithread*CLPAD]; iret++) {\r\n delete (T*)retiredObjects[ithread*CLPAD][iret];\r\n }\r\n delete[] hp[ithread*CLPAD];\r\n delete[] retiredObjects[ithread*CLPAD];\r\n }\r\n }\r\n\r\n\r\n /**\r\n * Progress Condition: wait-free bounded (by maxHPs)\r\n *\r\n * It's ok to use relaxed loads here because:\r\n * - For progress: we know that the store will eventually become visible,\r\n * or another publish() will take its place;\r\n * - For correctness: it can be re-ordered below, but at most it will protect\r\n * an object for longer than required, i.e. until the next publish overwrites it.\r\n * Or it gets re-ordered above, but only up to a seq-cst store on the same\r\n * variable in publish(), which _must_ be it, even if the store in the publish\r\n * is a release store (which is the case for publishRelease()).\r\n */\r\n void clear(const int tid) {\r\n for (int ihp = 0; ihp < maxHPs; ihp++) {\r\n hp[tid*CLPAD][ihp].store(nullptr, std::memory_order_relaxed);\r\n }\r\n }\r\n\r\n\r\n /**\r\n * Progress Condition: wait-free population oblivious\r\n */\r\n void clearOne(int ihp, const int tid) {\r\n hp[tid*CLPAD][ihp].store(nullptr,std::memory_order_relaxed);\r\n }\r\n\r\n\r\n /**\r\n * Progress Condition: lock-free\r\n */\r\n T* protect(int index, const std::atomic& atom, const int tid) {\r\n T* n = nullptr;\r\n T* ret;\r\n\t\twhile ((ret = atom.load()) != n) {\r\n\t\t\thp[tid*CLPAD][index].store(ret);\r\n\t\t\tn = ret;\r\n\t\t}\r\n\t\treturn ret;\r\n }\r\n\r\n inline T* get_protected(int index, const std::atomic& atom, const int tid) {\r\n return protect(index, atom, tid);\r\n }\r\n\r\n /**\r\n * This returns the same value that is passed as ptr, which is sometimes useful\r\n * Progress Condition: wait-free population oblivious\r\n */\r\n T* protectPtr(int index, T* ptr, const int tid) {\r\n hp[tid*CLPAD][index].store(ptr);\r\n return ptr;\r\n }\r\n\r\n /**\r\n * This assumes that the ptr lhead is already protected by a \"regular\" hazard pointers\r\n */\r\n void protectHead(int combinedIndex, T* lhead) {\r\n heads[combinedIndex*CLPAD].store(lhead, std::memory_order_release);\r\n }\r\n\r\n std::atomic* getHeads() {\r\n return heads;\r\n }\r\n\r\n /**\r\n * This returns the same value that is passed as ptr, which is sometimes useful\r\n * Progress Condition: wait-free population oblivious\r\n */\r\n T* protectRelease(int index, T* ptr, const int tid) {\r\n hp[tid*CLPAD][index].store(ptr, std::memory_order_release);\r\n return ptr;\r\n }\r\n\r\n /**\r\n * This returns the same value that is passed as ptr, which is sometimes useful\r\n * Progress Condition: wait-free bounded (by the number of threads squared)\r\n */\r\n void retire(T* ptr, const int tid) {\r\n if (numRetiredObjects[tid*CLPAD] >= HP_THRESHOLD_R) scanAndDelete(tid);\r\n retiredObjects[tid*CLPAD][numRetiredObjects[tid*CLPAD]++] = ptr;\r\n }\r\n\r\n\r\n void copyPtr(int index, int other, const int tid) {\r\n auto ptr = hp[tid*CLPAD][other].load(std::memory_order_relaxed);\r\n hp[tid*CLPAD][index].store(ptr, std::memory_order_release);\r\n }\r\n\r\n\r\nprivate:\r\n void scanAndDelete(const int tid) {\r\n for (int iret = 0; iret < numRetiredObjects[tid*CLPAD]; ) {\r\n bool ptrInUse = false;\r\n auto ptr = (T*)retiredObjects[tid*CLPAD][iret];\r\n for (int it = 0; it < maxThreads; it++) {\r\n for (int ihp = maxHPs-1; ihp >= 0; ihp--) {\r\n if (ptr == hp[it*CLPAD][ihp].load()) ptrInUse = true;\r\n }\r\n }\r\n if (ptrInUse) { iret++; continue; }\r\n // Scan the array of heads before deleting the pointer\r\n for (int icomb = 0; icomb < 2*MAX_THREADS; icomb++) {\r\n if (ptr == heads[icomb*CLPAD].load()) ptrInUse = true;\r\n }\r\n if (ptrInUse) { iret++; continue; }\r\n for (int i = iret; i < numRetiredObjects[tid*CLPAD]-1; i++) retiredObjects[tid*CLPAD][i] = retiredObjects[tid*CLPAD][i+1];\r\n numRetiredObjects[tid*CLPAD]--;\r\n delete ptr;\r\n\r\n }\r\n }\r\n};\r\n\r\n#endif /* _HAZARD_POINTERS_H_ */\r\n" }, { "path": "common/HazardPointersSimQueue.hpp", "content": "/******************************************************************************\r\n * Copyright (c) 2014-2016, Pedro Ramalhete, Andreia Correia\r\n * All rights reserved.\r\n *\r\n * Redistribution and use in source and binary forms, with or without\r\n * modification, are permitted provided that the following conditions are met:\r\n * * Redistributions of source code must retain the above copyright\r\n * notice, this list of conditions and the following disclaimer.\r\n * * Redistributions in binary form must reproduce the above copyright\r\n * notice, this list of conditions and the following disclaimer in the\r\n * documentation and/or other materials provided with the distribution.\r\n * * Neither the name of Concurrency Freaks nor the\r\n * names of its contributors may be used to endorse or promote products\r\n * derived from this software without specific prior written permission.\r\n\r\n *\r\n * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\r\n * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED\r\n * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\r\n * DISCLAIMED. IN NO EVENT SHALL BE LIABLE FOR ANY\r\n * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\r\n * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\r\n * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND\r\n * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\r\n * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\r\n * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\r\n ******************************************************************************\r\n */\r\n\r\n#ifndef _HAZARD_POINTERS_SIM_QUEUE_H_\r\n#define _HAZARD_POINTERS_SIM_QUEUE_H_\r\n\r\n#include \r\n#include \r\n#include \r\n#include \r\n\r\n\r\n/*\r\n * The main difference from this implementation to regular Hazard Pointers is\r\n * that the constructor takes a function pointer to function 'find' which\r\n * acts as a callback, returning true if the pointer is still stored somewhere\r\n * in the data structure. This is used by SimQueue to indicate if there is a\r\n * pointer to the object we're trying to de-allocate in the array of enqReused.\r\n */\r\ntemplate\r\nclass HazardPointersSimQueue {\r\n\r\nprivate:\r\n static const int HP_MAX_THREADS = 128;\r\n static const int HP_MAX_HPS = 11; // This is named 'K' in the HP paper\r\n static const int CLPAD = 128/sizeof(std::atomic);\r\n static const int HP_THRESHOLD_R = 0; // This is named 'R' in the HP paper\r\n static const int MAX_RETIRED = HP_MAX_THREADS*HP_MAX_HPS; // Maximum number of retired objects per thread\r\n\r\n const int maxHPs;\r\n const int maxThreads;\r\n\r\n std::atomic hp[HP_MAX_THREADS*CLPAD][HP_MAX_HPS];\r\n // It's not nice that we have a lot of empty vectors, but we need padding to avoid false sharing\r\n std::vector retiredList[HP_MAX_THREADS*CLPAD];\r\n\r\n std::function findPtr;\r\n\r\npublic:\r\n HazardPointersSimQueue(std::function& find, int maxHPs=HP_MAX_HPS, int maxThreads=HP_MAX_THREADS) : maxHPs{maxHPs}, maxThreads{maxThreads} {\r\n findPtr = find;\r\n for (int ithread = 0; ithread < HP_MAX_THREADS; ithread++) {\r\n for (int ihp = 0; ihp < HP_MAX_HPS; ihp++) {\r\n hp[ithread*CLPAD][ihp].store(nullptr, std::memory_order_relaxed);\r\n }\r\n }\r\n }\r\n\r\n ~HazardPointersSimQueue() {\r\n for (int ithread = 0; ithread < HP_MAX_THREADS; ithread++) {\r\n // Clear the current retired nodes\r\n for (unsigned iret = 0; iret < retiredList[ithread*CLPAD].size(); iret++) {\r\n delete retiredList[ithread*CLPAD][iret];\r\n }\r\n }\r\n }\r\n\r\n\r\n /**\r\n * Progress Condition: wait-free bounded (by maxHPs)\r\n */\r\n void clear(const int tid) {\r\n for (int ihp = 0; ihp < maxHPs; ihp++) {\r\n hp[tid*CLPAD][ihp].store(nullptr, std::memory_order_release);\r\n }\r\n }\r\n\r\n\r\n /**\r\n * Progress Condition: wait-free population oblivious\r\n */\r\n void clearOne(int ihp, const int tid) {\r\n hp[tid*CLPAD][ihp].store(nullptr, std::memory_order_release);\r\n }\r\n\r\n\r\n /**\r\n * Progress Condition: lock-free\r\n */\r\n T* protect(int index, const std::atomic& atom, const int tid) {\r\n T* n = nullptr;\r\n T* ret;\r\n\t\twhile ((ret = atom.load()) != n) {\r\n\t\t\thp[tid*CLPAD][index].store(ret);\r\n\t\t\tn = ret;\r\n\t\t}\r\n\t\treturn ret;\r\n }\r\n\r\n /**\r\n * This returns the same value that is passed as ptr, which is sometimes useful\r\n * Progress Condition: wait-free population oblivious\r\n */\r\n T* protectPtr(int index, T* ptr, const int tid) {\r\n hp[tid*CLPAD][index].store(ptr);\r\n return ptr;\r\n }\r\n\r\n\r\n /**\r\n * This returns the same value that is passed as ptr, which is sometimes useful\r\n * Progress Condition: wait-free population oblivious\r\n */\r\n T* protectRelease(int index, T* ptr, const int tid) {\r\n hp[tid*CLPAD][index].store(ptr, std::memory_order_release);\r\n return ptr;\r\n }\r\n\r\n\r\n /**\r\n * This returns the same value that is passed as ptr, which is sometimes useful\r\n * Progress Condition: wait-free bounded (by the number of threads squared)\r\n */\r\n void retire(T* ptr, const int tid) {\r\n retiredList[tid*CLPAD].push_back(ptr);\r\n for (unsigned iret = 0; iret < retiredList[tid*CLPAD].size();) {\r\n auto obj = retiredList[tid*CLPAD][iret];\r\n if (findPtr(obj)) {\r\n iret++;\r\n continue;\r\n }\r\n bool canDelete = true;\r\n for (int tid = 0; tid < maxThreads && canDelete; tid++) {\r\n for (int ihp = maxHPs-1; ihp >= 0; ihp--) {\r\n if (hp[tid*CLPAD][ihp].load() == obj) {\r\n canDelete = false;\r\n break;\r\n }\r\n }\r\n }\r\n if (canDelete) {\r\n retiredList[tid*CLPAD].erase(retiredList[tid*CLPAD].begin() + iret);\r\n delete obj;\r\n continue;\r\n }\r\n iret++;\r\n }\r\n }\r\n};\r\n\r\n#endif /* _HAZARD_POINTERS_H_ */\r\n" }, { "path": "common/README.md", "content": "Here are some files that are needed by other libraries and data structures:\r\n\r\n HazardEras.hpp Used by some of the lock-free data structures for memory reclamation\r\n HazardPointers.hpp Used by some of the lock-free data structures for memory reclamation\r\n HazardPointersSimQueue.hpp Used by SimQueue for memory reclamation. Notice that the original SimQueue implementation in C does not ha memory reclamation. This implementation in C++ with this modified version of Hazard Pointers was done by Correia and Ramalhete\r\n pfences.h Used by Romulus\r\n RIStaticPerThread.hpp Used by Romulus\r\n ThreadRegistry.cpp Used by Romulus\r\n ThreadRegistry.hpp Used by Romulus" }, { "path": "common/RIStaticPerThread.hpp", "content": "/******************************************************************************\r\n * Copyright (c) 2014-2016, Pedro Ramalhete, Andreia Correia\r\n * All rights reserved.\r\n *\r\n * Redistribution and use in source and binary forms, with or without\r\n * modification, are permitted provided that the following conditions are met:\r\n * * Redistributions of source code must retain the above copyright\r\n * notice, this list of conditions and the following disclaimer.\r\n * * Redistributions in binary form must reproduce the above copyright\r\n * notice, this list of conditions and the following disclaimer in the\r\n * documentation and/or other materials provided with the distribution.\r\n * * Neither the name of Concurrency Freaks nor the\r\n * names of its contributors may be used to endorse or promote products\r\n * derived from this software without specific prior written permission.\r\n\r\n *\r\n * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\r\n * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED\r\n * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\r\n * DISCLAIMED. IN NO EVENT SHALL BE LIABLE FOR ANY\r\n * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\r\n * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\r\n * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND\r\n * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\r\n * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\r\n * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\r\n ******************************************************************************\r\n */\r\n\r\n#ifndef _RISTATIC_H_\r\n#define _RISTATIC_H_\r\n\r\n#include \r\n#include \r\n#include \r\n\r\n\r\n// TODO: change all calls that need the tid to use a function argument\r\n// TODO: use std::vector instead of arrays for the retired objects (keep the padding)\r\nclass RIStaticPerThread {\r\n\r\nprivate:\r\n const int maxThreads;\r\n alignas(128) std::atomic* states;\r\n\r\n static const uint64_t NOT_READING = 0;\r\n static const uint64_t READING = 1;\r\n static const int CLPAD = 128/sizeof(uint64_t);\r\n\r\npublic:\r\n RIStaticPerThread(int maxThreads) : maxThreads{maxThreads} {\r\n states = new std::atomic[maxThreads*CLPAD];\r\n for (int tid = 0; tid < maxThreads; tid++) {\r\n states[tid*CLPAD].store(NOT_READING, std::memory_order_relaxed);\r\n }\r\n }\r\n\r\n ~RIStaticPerThread() {\r\n delete[] states;\r\n }\r\n\r\n // Will attempt to pass all current READING states to\r\n inline void abortRollback() noexcept {\r\n for (int tid = 0; tid < maxThreads; tid++) {\r\n if (states[tid*CLPAD].load() != READING) continue;\r\n uint64_t read = READING;\r\n states[tid*CLPAD].compare_exchange_strong(read, READING+1);\r\n }\r\n }\r\n\r\n // Returns true if the arrival was successfully rollbacked.\r\n // If there was a writer changing the state to READING+1 then it will\r\n // return false, meaning that the arrive() is still valid and visible.\r\n inline bool rollbackArrive(const int tid) noexcept {\r\n return (states[tid*CLPAD].fetch_add(-1) == READING);\r\n }\r\n\r\n inline void arrive(const int tid) noexcept {\r\n states[tid*CLPAD].store(READING);\r\n }\r\n\r\n inline void depart(const int tid) noexcept {\r\n states[tid*CLPAD].store(NOT_READING); // Making this \"memory_order_release\" will cause overflows!\r\n }\r\n\r\n inline bool isEmpty() noexcept {\r\n for (int tid = 0; tid < maxThreads; tid++) {\r\n if (states[tid*CLPAD].load() != NOT_READING) return false;\r\n }\r\n return true;\r\n }\r\n};\r\n\r\n#endif /* RISTATIC_H */\r\n" }, { "path": "common/ThreadRegistry.cpp", "content": "/*\r\n * Contains all global variables.\r\n */\r\n#include \"common/ThreadRegistry.hpp\"\r\n\r\n// Global/singleton to hold all the thread registry functionality\r\nThreadRegistry gThreadRegistry {};\r\n\r\n// This is where every thread stores the tid it has been assigned when it calls getTID() for the first time.\r\n// When the thread dies, the destructor of ThreadCheckInCheckOut will be called and de-register the thread.\r\nthread_local ThreadCheckInCheckOut tl_tcico {};\r\n\r\nvoid thread_registry_deregister_thread(const int tid) {\r\n gThreadRegistry.deregister_thread(tid);\r\n}\r\n" }, { "path": "common/ThreadRegistry.hpp", "content": "#ifndef _THREAD_REGISTRY_H_\r\n#define _THREAD_REGISTRY_H_\r\n\r\n#include \r\n#include \r\n#include \r\n#include \r\n\r\n// Increase this if 128 threads is not enough\r\nstatic const int REGISTRY_MAX_THREADS = 128;\r\n\r\n\r\nextern void thread_registry_deregister_thread(const int tid);\r\n\r\n\r\n// An helper class to do the checkin and checkout of the thread registry\r\nstruct ThreadCheckInCheckOut {\r\n static const int NOT_ASSIGNED = -1;\r\n int tid { NOT_ASSIGNED };\r\n ~ThreadCheckInCheckOut() {\r\n if (tid == NOT_ASSIGNED) return;\r\n thread_registry_deregister_thread(tid);\r\n }\r\n};\r\n\r\n\r\nextern thread_local ThreadCheckInCheckOut tl_tcico;\r\n\r\n\r\n// Forward declaration of global/singleton instance\r\nclass ThreadRegistry;\r\nextern ThreadRegistry gThreadRegistry;\r\n\r\n\r\n/*\r\n *

Registry for threads

\r\n *\r\n * This is singleton type class that allows assignement of a unique id to each thread.\r\n * The first time a thread calls ThreadRegistry::getTID() it will allocate a free slot in 'usedTID[]'.\r\n * This tid wil be saved in a thread-local variable of the type ThreadCheckInCheckOut which\r\n * upon destruction of the thread will call the destructor of ThreadCheckInCheckOut and free the\r\n * corresponding slot to be used by a later thread.\r\n * RomulusLR relies on this to work properly.\r\n */\r\nclass ThreadRegistry {\r\nprivate:\r\n alignas(128) std::atomic usedTID[REGISTRY_MAX_THREADS]; // Which TIDs are in use by threads\r\n alignas(128) std::atomic maxTid {-1}; // Highest TID (+1) in use by threads\r\n\r\npublic:\r\n ThreadRegistry() {\r\n for (int it = 0; it < REGISTRY_MAX_THREADS; it++) {\r\n usedTID[it].store(false, std::memory_order_relaxed);\r\n }\r\n }\r\n\r\n /*\r\n * Progress Condition: wait-free bounded (by the number of threads)\r\n */\r\n int register_thread_new(void) {\r\n for (int tid = 0; tid < REGISTRY_MAX_THREADS; tid++) {\r\n if (usedTID[tid].load(std::memory_order_acquire)) continue;\r\n bool unused = false;\r\n if (!usedTID[tid].compare_exchange_strong(unused, true)) continue;\r\n // Increase the current maximum to cover our thread id\r\n int curMax = maxTid.load();\r\n while (curMax <= tid) {\r\n maxTid.compare_exchange_strong(curMax, tid+1);\r\n curMax = maxTid.load();\r\n }\r\n tl_tcico.tid = tid;\r\n return tid;\r\n }\r\n std::cout << \"ERROR: Too many threads, registry can only hold \" << REGISTRY_MAX_THREADS << \" threads\\n\";\r\n assert(false);\r\n }\r\n\r\n /*\r\n * Progress condition: wait-free population oblivious\r\n */\r\n inline void deregister_thread(const int tid) {\r\n usedTID[tid].store(false, std::memory_order_release);\r\n }\r\n\r\n /*\r\n * Progress condition: wait-free population oblivious\r\n */\r\n static inline uint64_t getMaxThreads(void) {\r\n return gThreadRegistry.maxTid.load(std::memory_order_acquire);\r\n }\r\n\r\n /*\r\n * Progress condition: wait-free bounded (by the number of threads)\r\n */\r\n static inline int getTID(void) {\r\n int tid = tl_tcico.tid;\r\n if (tid != ThreadCheckInCheckOut::NOT_ASSIGNED) return tid;\r\n return gThreadRegistry.register_thread_new();\r\n }\r\n};\r\n\r\n#endif /* _THREAD_REGISTRY_H_ */\r\n" }, { "path": "common/pfences.h", "content": "/*\r\n * Copyright 2017-2018\r\n * Andreia Correia \r\n * Pedro Ramalhete \r\n * Pascal Felber \r\n * Nachshon Cohen \r\n *\r\n * This work is published under the MIT license. See LICENSE.txt\r\n */\r\n#ifndef _PERSISTENT_FENCES_\r\n#define _PERSISTENT_FENCES_\r\n\r\n/*\r\n * The naming for these macros and respective operations were taken from the excellent\r\n * \"Preserving Happens-before in Persistent Memory\" by Izraelevitz, Mendes, and Scott\r\n * https://www.cs.rochester.edu/u/jhi1/papers/2016-spaa-transform\r\n *\r\n * We have five different definitions of pwb/pfence/psync:\r\n * - Emulated: We introduce a delay on stores, like Mnemosyne does\r\n * - Nothing: only works with process restart persistency, i.e. process failures, but not system failure\r\n * - Define pwb as clflush (Broadwell cpus)\r\n * - Define pwb as clflushopt (most x86 cpus)\r\n * - Define pwb as clwb (only very recent cpus have this instruction)\r\n */\r\n\r\n/*\r\n * We copied the methods from Menmosyne:\r\n * http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.221.5462&rep=rep1&type=pdf\r\n */\r\nstatic inline unsigned long long asm_rdtsc(void)\r\n{\r\n unsigned hi, lo;\r\n __asm__ __volatile__ (\"rdtsc\" : \"=a\"(lo), \"=d\"(hi));\r\n return ( (unsigned long long)lo)|( ((unsigned long long)hi)<<32 );\r\n}\r\n\r\n// Change this depending on the clock cycle of your cpu. For Cervino it's 2100, for my laptop it's 2712.\r\n#define EMULATED_CPUFREQ 2100\r\n\r\n#define NS2CYCLE(__ns) ((__ns) * EMULATED_CPUFREQ / 1000)\r\n\r\nstatic inline void emulate_latency_ns(int ns) {\r\n uint64_t stop;\r\n uint64_t start = asm_rdtsc();\r\n uint64_t cycles = NS2CYCLE(ns);\r\n do {\r\n /* RDTSC doesn't necessarily wait for previous instructions to complete\r\n * so a serializing instruction is usually used to ensure previous\r\n * instructions have completed. However, in our case this is a desirable\r\n * property since we want to overlap the latency we emulate with the\r\n * actual latency of the emulated instruction.\r\n */\r\n stop = asm_rdtsc();\r\n } while (stop - start < cycles);\r\n}\r\n\r\n\r\n/*\r\n * We use the settings on the delays for emulation from the NVMOVE paper:\r\n * http://www.cs.utexas.edu/~vijay/papers/inflow16-nvmove.pdf\r\n */\r\n#ifdef PWB_IS_STT\r\n /* Delays for emulating STT in DRAM */\r\n #define PWB(addr) emulate_latency_ns(140)\r\n #define PFENCE() emulate_latency_ns(200)\r\n #define PSYNC() emulate_latency_ns(200)\r\n#elif PWB_IS_PCM\r\n /* Delays for emulating PCM in DRAM */\r\n #define PWB(addr) emulate_latency_ns(340)\r\n #define PFENCE() emulate_latency_ns(500)\r\n #define PSYNC() emulate_latency_ns(500)\r\n#elif PWB_IS_CLFLUSH\r\n /*\r\n * More info at http://elixir.free-electrons.com/linux/latest/source/arch/x86/include/asm/special_insns.h#L213\r\n * Intel programming manual at https://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-optimization-manual.pdf\r\n * Use these for Broadwell CPUs (cervino server)\r\n */\r\n #define PWB(addr) __asm__ volatile(\"clflush (%0)\" :: \"r\" (addr) : \"memory\") // Broadwell only works with this.\r\n #define PFENCE() {} // No ordering fences needed for CLFLUSH (section 7.4.6 of Intel manual)\r\n #define PSYNC() {} // For durability it's not obvious, but CLFLUSH seems to be enough, and PMDK uses the same approach\r\n#elif PWB_IS_CLWB\r\n /* Use this for CPUs that support clwb, such as the SkyLake SP series (c5 compute intensive instances in AWS are an example of it) */\r\n #define PWB(addr) __asm__ volatile(\".byte 0x66; xsaveopt %0\" : \"+m\" (*(volatile char *)(addr))) // clwb() only for Ice Lake onwards\r\n #define PFENCE() __asm__ volatile(\"sfence\" : : : \"memory\")\r\n #define PSYNC() __asm__ volatile(\"sfence\" : : : \"memory\")\r\n#elif PWB_IS_NOP\r\n /* pwbs are not needed for shared memory persistency (i.e. persistency across process failure) */\r\n #define PWB(addr) {}\r\n #define PFENCE() __asm__ volatile(\"sfence\" : : : \"memory\")\r\n #define PSYNC() __asm__ volatile(\"sfence\" : : : \"memory\")\r\n#elif PWB_IS_CLFLUSHOPT\r\n /* Use this for CPUs that support clflushopt, which is most recent x86 */\r\n #define PWB(addr) __asm__ volatile(\".byte 0x66; clflush %0\" : \"+m\" (*(volatile char *)(addr))) // clflushopt (Kaby Lake)\r\n #define PFENCE() __asm__ volatile(\"sfence\" : : : \"memory\")\r\n #define PSYNC() __asm__ volatile(\"sfence\" : : : \"memory\")\r\n#else\r\n#error \"You must define what PWB is. Choose PWB_IS_CLFLUSHOPT if you don't know what your CPU is capable of\"\r\n#endif\r\n\r\n// Flush each cache line in a range\r\n// TODO: fix cache alignment\r\ninline static void flushFromTo(void* from, void* to) noexcept {\r\n const int cache_line_size = 64;\r\n uint8_t* ptr = (uint8_t*)from;\r\n for (; ptr < (uint8_t*)to; ptr += cache_line_size) PWB(ptr);\r\n}\r\n\r\n// TODO: Implement fences for ARM\r\n\r\n\r\n#endif\r\n" }, { "path": "datastructures/generic/TMHashMap.hpp", "content": "/******************************************************************************\r\n * Copyright (c) 2014-2018, Pedro Ramalhete, Andreia Correia\r\n * All rights reserved.\r\n *\r\n * Redistribution and use in source and binary forms, with or without\r\n * modification, are permitted provided that the following conditions are met:\r\n * * Redistributions of source code must retain the above copyright\r\n * notice, this list of conditions and the following disclaimer.\r\n * * Redistributions in binary form must reproduce the above copyright\r\n * notice, this list of conditions and the following disclaimer in the\r\n * documentation and/or other materials provided with the distribution.\r\n * * Neither the name of Concurrency Freaks nor the\r\n * names of its contributors may be used to endorse or promote products\r\n * derived from this software without specific prior written permission.\r\n *\r\n * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\r\n * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED\r\n * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\r\n * DISCLAIMED. IN NO EVENT SHALL BE LIABLE FOR ANY\r\n * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\r\n * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\r\n * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND\r\n * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\r\n * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\r\n * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\r\n ******************************************************************************\r\n */\r\n\r\n#ifndef _TM_NON_RESIZABLE_HASH_MAP_H_\r\n#define _TM_NON_RESIZABLE_HASH_MAP_H_\r\n\r\n#include \r\n#include \r\n#include \r\n\r\n#include \"../../stms/tm.h\" // This header defines the macros for the STM being compiled\r\n\r\n\r\n/**\r\n *

A Non-Resizable Hash Map for usage with STMs

\r\n *\r\n * Each node contains 4 entries (key/value) so as to provide better cache locality\r\n *\r\n *\r\n * TODO\r\n\r\n *\r\n * @author Pedro Ramalhete\r\n * @author Andreia Correia\r\n */\r\ntemplate\r\nclass TMHashMap : public TM_BASE_TYPE {\r\n\r\nprivate:\r\n // One KeyVal is 16+16 bytes, therefore, 4 KeyVals are 2 cache lines in x86 (128 bytes)\r\n static const int KV_NUM = 4;\r\n static const unsigned int MAX_THREADS = 128;\r\n const unsigned int maxThreads;\r\n const unsigned int capacity;\r\n\r\n struct KeyVal {\r\n //uint64_t h; // Full hash of the key, for faster comparison. TODO: add code to handle h\r\n TM_TYPE key {nullptr};\r\n TM_TYPE val {nullptr};\r\n KeyVal() {}\r\n KeyVal(K* key, V* value) : key{key}, val{value} { }\r\n };\r\n\r\n struct Node : TM_BASE_TYPE {\r\n KeyVal kv[KV_NUM];\r\n TM_TYPE next {nullptr};\r\n Node() {}\r\n Node(K* key, V* value) {\r\n kv[0].key = key;\r\n kv[0].val = value;\r\n }\r\n bool isEmpty() {\r\n for (int i = 0; i < KV_NUM; i++) {\r\n if (kv[i].key != nullptr) return false;\r\n }\r\n return true;\r\n }\r\n };\r\n\r\n alignas(128) Node* buckets; // An array of Nodes\r\n\r\n\r\n int myhash(K* key) { return 0; } // Used only for tests\r\n\r\npublic:\r\n TMHashMap(unsigned int maxThreads=MAX_THREADS, unsigned int capacity=2*1024*1024) : maxThreads{maxThreads}, capacity{capacity} {\r\n buckets = new Node[capacity];\r\n }\r\n\r\n\r\n ~TMHashMap() {\r\n delete[] buckets;\r\n }\r\n\r\n\r\n std::string className() { return TM_NAME() + \"-HashMap\"; }\r\n\r\n\r\n /*\r\n * Progress Condition: lock-free\r\n * Adds a node with a key if the key is not present, otherwise replaces the value.\r\n * Returns the previous value (nullptr by default).\r\n */\r\n V* put(K* key, V* value, const int tid) {\r\n if (key == nullptr) throw std::invalid_argument(\"key can not be nullptr\");\r\n if (value == nullptr) throw std::invalid_argument(\"value can not be nullptr\");\r\n V* oldVal = nullptr;\r\n KeyVal *firstFree = nullptr;\r\n auto h = std::hash{}(*key);\r\n Node* node = &buckets[h];\r\n while (true) {\r\n for (int i = 0; i < KV_NUM; i++) {\r\n KeyVal& kv = node->kv[i];\r\n if (kv.key == nullptr) {\r\n // Save the first available entry, in case we need to insert somewhere\r\n if (firstFree == nullptr) firstFree = &kv;\r\n continue;\r\n }\r\n if (*kv.key != *key) continue;\r\n // Found a matching key, replace the old value with the new\r\n oldVal = kv.val;\r\n kv.val = value;\r\n return oldVal;\r\n }\r\n Node* lnext = node->next;\r\n if (lnext == nullptr) break;\r\n node = lnext;\r\n }\r\n // We got here without a replacement, so insert in the first available\r\n if (firstFree == nullptr) {\r\n // No available entry, allocate a node and insert it there\r\n Node* newNode = TM_ALLOC(key,value);\r\n node->next = newNode;\r\n } else {\r\n firstFree->key = key;\r\n firstFree->val = value;\r\n }\r\n return oldVal;\r\n }\r\n\r\n\r\n /*\r\n * Progress Condition: lock-free\r\n * Removes a key, returning the value associated with it.\r\n * Returns nullptr if there is no matching key.\r\n */\r\n V* removeKey(K* key, const int tid) {\r\n if (key == nullptr) throw std::invalid_argument(\"key can not be nullptr\");\r\n auto h = std::hash{}(*key);\r\n Node* node = &buckets[h];\r\n Node* prev = node;\r\n while (true) {\r\n for (int i = 0; i < KV_NUM; i++) {\r\n KeyVal& kv = node->kv[i];\r\n if (kv.key == nullptr || *kv.key != *key) continue;\r\n // Found a matching key, replace the old value with nullptr\r\n V* oldVal = kv.val;\r\n kv.val = nullptr;\r\n kv.key = nullptr;\r\n // Check if it's the first node and if it is empty, then unlink it and free it\r\n if (prev != node && node->isEmpty()) {\r\n prev->next = node->next;\r\n TM_FREE(node);\r\n }\r\n return oldVal;\r\n }\r\n prev = node;\r\n node = node->next;\r\n // We got to the end without a matching key, return nullptr\r\n if (node == nullptr) return nullptr;\r\n }\r\n }\r\n\r\n\r\n /*\r\n * Progress Condition: lock-free\r\n * Returns the value of associated with the key, nullptr if there is no mapping\r\n */\r\n V* get(K* key, const int tid) {\r\n if (key == nullptr) throw std::invalid_argument(\"key can not be nullptr\");\r\n auto h = std::hash{}(*key);\r\n Node* node = &buckets[h];\r\n while (true) {\r\n for (int i = 0; i < KV_NUM; i++) {\r\n KeyVal& kv = node->kv[i];\r\n if (kv.key == nullptr || *kv.key != *key) continue;\r\n return kv.val;\r\n }\r\n Node* lnext = node->next;\r\n if (lnext == nullptr) return nullptr;\r\n node = lnext;\r\n }\r\n }\r\n\r\n\r\n\r\n //\r\n // Set methods for running the usual tests and benchmarks\r\n //\r\n\r\n bool add(K* key, const int tid) {\r\n return TM_WRITE_TRANSACTION([&] () -> bool {\r\n return put(key,key, tid) == nullptr;\r\n });\r\n }\r\n\r\n bool remove(K* key, const int tid) {\r\n return TM_WRITE_TRANSACTION([this,key,tid] () -> bool {\r\n return removeKey(key, tid) != nullptr;\r\n });\r\n }\r\n\r\n bool contains(K* key, const int tid) {\r\n return TM_READ_TRANSACTION([this,key,tid] () -> bool {\r\n return get(key, tid) != nullptr;\r\n });\r\n }\r\n\r\n // Used only for benchmarks. It's single-threaded\r\n bool addAll(K** keys, const int size, const int tid) {\r\n for (int i = 0; i < size; i++) add(keys[i], tid);\r\n }\r\n\r\n};\r\n\r\n#endif /* _TM_NON_RESIZABLE_HASH_MAP_H_ */\r\n" }, { "path": "datastructures/generic/TMLinkedListQueue.hpp", "content": "/******************************************************************************\r\n * Copyright (c) 2014-2017, Pedro Ramalhete, Andreia Correia\r\n * All rights reserved.\r\n *\r\n * Redistribution and use in source and binary forms, with or without\r\n * modification, are permitted provided that the following conditions are met:\r\n * * Redistributions of source code must retain the above copyright\r\n * notice, this list of conditions and the following disclaimer.\r\n * * Redistributions in binary form must reproduce the above copyright\r\n * notice, this list of conditions and the following disclaimer in the\r\n * documentation and/or other materials provided with the distribution.\r\n * * Neither the name of Concurrency Freaks nor the\r\n * names of its contributors may be used to endorse or promote products\r\n * derived from this software without specific prior written permission.\r\n *\r\n * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\r\n * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED\r\n * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\r\n * DISCLAIMED. IN NO EVENT SHALL BE LIABLE FOR ANY\r\n * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\r\n * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\r\n * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND\r\n * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\r\n * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\r\n * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\r\n ******************************************************************************\r\n */\r\n\r\n#ifndef _TM_LINKED_LIST_QUEUE_H_\r\n#define _TM_LINKED_LIST_QUEUE_H_\r\n\r\n#include \r\n#include \r\n\r\n#include \"../../stms/CRWWPSTM.hpp\"\r\n#include \"../../stms/LeftRightTM.hpp\"\r\n#include \"../../stms/tm.h\" // This header defines the macros for the STM being compiled\r\n#include \"MWCLF.hpp\"\r\n#include \"MWCWF.hpp\"\r\n#include \"CXTM.hpp\"\r\n\r\n/**\r\n *

A Linked List queue using STM

\r\n *\r\n *\r\n * TODO\r\n *\r\n *\r\n * enqueue algorithm: sequential implementation + MWC\r\n * dequeue algorithm: sequential implementation + MWC\r\n * Consistency: Linearizable\r\n * enqueue() progress: lock-free\r\n * dequeue() progress: lock-free\r\n * Memory Reclamation: Hazard Eras (integrated into MWC)\r\n * enqueue min ops: 2 DCAS + 1 CAS\r\n * dequeue min ops: 1 DCAS + 1 CAS\r\n *\r\n * @author Pedro Ramalhete\r\n * @author Andreia Correia\r\n */\r\ntemplate\r\nclass TMLinkedListQueue {\r\n\r\nprivate:\r\n static const unsigned int MAX_THREADS = 128;\r\n const unsigned int maxThreads;\r\n\r\n struct Node : TM_BASE_TYPE {\r\n T* item;\r\n TM_TYPE next;\r\n Node(T* userItem) : item{userItem}, next{nullptr} { }\r\n };\r\n\r\n alignas(128) TM_TYPE head {nullptr};\r\n alignas(128) TM_TYPE tail {nullptr};\r\n\r\n\r\npublic:\r\n TMLinkedListQueue(unsigned int maxThreads=MAX_THREADS) : maxThreads{maxThreads} {\r\n Node* sentinelNode = TM_ALLOC(nullptr);\r\n head = sentinelNode;\r\n tail = sentinelNode;\r\n }\r\n\r\n\r\n ~TMLinkedListQueue() {\r\n // TODO: replace this 0 with the actual tid otherwise we could have issues\r\n while (dequeue(0) != nullptr); // Drain the queue\r\n Node* lhead = head;\r\n delete lhead;\r\n }\r\n\r\n\r\n static std::string className() { return TM_NAME() + \"-LinkedListQueue\"; }\r\n\r\n\r\n /*\r\n *\r\n * Always returns true\r\n */\r\n bool enqueue(T* item, const int tid) {\r\n if (item == nullptr) throw std::invalid_argument(\"item can not be nullptr\");\r\n return TM_WRITE_TRANSACTION([this,item] () -> bool {\r\n Node* newNode = TM_ALLOC(item);\r\n tail->next = newNode;\r\n tail = newNode;\r\n return true;\r\n });\r\n }\r\n\r\n\r\n /*\r\n *\r\n */\r\n T* dequeue(const int tid) {\r\n return TM_WRITE_TRANSACTION([this] () -> T* {\r\n Node* lhead = head;\r\n if (lhead == tail) return nullptr;\r\n head = lhead->next;\r\n TM_FREE(lhead);\r\n return head->item;\r\n });\r\n }\r\n};\r\n\r\n#endif /* _MWC_LINKED_LIST_QUEUE_H_ */\r\n" }, { "path": "datastructures/generic/TMLinkedListSet.hpp", "content": "/******************************************************************************\r\n * Copyright (c) 2014-2018, Pedro Ramalhete, Andreia Correia\r\n * All rights reserved.\r\n *\r\n * Redistribution and use in source and binary forms, with or without\r\n * modification, are permitted provided that the following conditions are met:\r\n * * Redistributions of source code must retain the above copyright\r\n * notice, this list of conditions and the following disclaimer.\r\n * * Redistributions in binary form must reproduce the above copyright\r\n * notice, this list of conditions and the following disclaimer in the\r\n * documentation and/or other materials provided with the distribution.\r\n * * Neither the name of Concurrency Freaks nor the\r\n * names of its contributors may be used to endorse or promote products\r\n * derived from this software without specific prior written permission.\r\n *\r\n * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\r\n * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED\r\n * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\r\n * DISCLAIMED. IN NO EVENT SHALL BE LIABLE FOR ANY\r\n * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\r\n * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\r\n * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND\r\n * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\r\n * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\r\n * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\r\n ******************************************************************************\r\n */\r\n\r\n#ifndef _TM_LINKED_LIST_SET_H_\r\n#define _TM_LINKED_LIST_SET_H_\r\n\r\n#include \r\n#include \r\n\r\n#include \"../../stms/tm.h\" // This header defines the macros for the STM being compiled\r\n\r\n\r\n/**\r\n *

A Linked List Set for usage with STMs

\r\n *\r\n * TODO\r\n *\r\n *\r\n * @author Pedro Ramalhete\r\n * @author Andreia Correia\r\n */\r\ntemplate\r\nclass TMLinkedListSet : public TM_BASE_TYPE {\r\n\r\nprivate:\r\n static const unsigned int MAX_THREADS = 128;\r\n const unsigned int maxThreads;\r\n\r\n struct Node : public TM_BASE_TYPE {\r\n T* key;\r\n TM_TYPE next;\r\n Node(T* key) : key{key}, next{nullptr} { }\r\n };\r\n\r\n alignas(128) TM_TYPE head {nullptr};\r\n alignas(128) TM_TYPE tail {nullptr};\r\n\r\n\r\npublic:\r\n TMLinkedListSet(unsigned int maxThreads=MAX_THREADS) : maxThreads{maxThreads} {\r\n Node* lhead = new Node(nullptr);\r\n Node* ltail = new Node(nullptr);\r\n head = lhead;\r\n head->next = ltail;\r\n tail = ltail;\r\n }\r\n\r\n\r\n ~TMLinkedListSet() {\r\n // Delete all the nodes in the list\r\n Node* prev = head;\r\n Node* node = prev->next;\r\n while (node != tail) {\r\n delete prev;\r\n prev = node;\r\n node = node->next;\r\n }\r\n delete prev;\r\n delete tail;\r\n }\r\n\r\n\r\n static std::string className() { return TM_NAME() + \"-LinkedListSet\"; }\r\n\r\n#ifdef TINY_STM\r\n /*\r\n * Progress Condition: lock-free\r\n * Adds a node with a key, returns false if the key is already in the set\r\n */\r\n bool add(T* key, const int tid) {\r\n if (key == nullptr) throw std::invalid_argument(\"key can not be nullptr\");\r\n bool retval = false;\r\n WRITE_TX_BEGIN\r\n Node* newNode = TM_ALLOC(key);\r\n Node* prev = head;\r\n Node* node = prev->next;\r\n while (true) {\r\n if (node == tail) {\r\n prev->next = newNode;\r\n newNode->next = node;\r\n retval = true;\r\n break;\r\n }\r\n if (*key == *node->key) {\r\n TM_FREE(newNode); // If the key was already in the set, free the node that was never used\r\n break;\r\n }\r\n if (*(node->key) < *key) {\r\n prev->next = newNode;\r\n newNode->next = node;\r\n retval = true;\r\n break;\r\n }\r\n prev = node;\r\n node = node->next;\r\n }\r\n WRITE_TX_END\r\n return retval;\r\n }\r\n\r\n\r\n /*\r\n * Progress Condition: lock-free\r\n * Removes a node with an key, returns false if the key is not in the set\r\n */\r\n bool remove(T* key, const int tid) {\r\n if (key == nullptr) throw std::invalid_argument(\"key can not be nullptr\");\r\n bool retval = false;\r\n WRITE_TX_BEGIN\r\n Node* prev = head;\r\n Node* node = prev->next;\r\n while (true) {\r\n if (node == tail) break;\r\n if (*key == *node->key) {\r\n prev->next = node->next;\r\n TM_FREE(node);\r\n retval = true;\r\n break;\r\n }\r\n if (*(node->key) < *key) break;\r\n prev = node;\r\n node = node->next;\r\n }\r\n WRITE_TX_END\r\n return retval;\r\n }\r\n\r\n\r\n /*\r\n * Progress Condition: lock-free\r\n * Returns true if it finds a node with a matching key\r\n */\r\n bool contains(T* key, const int tid) {\r\n if (key == nullptr) throw std::invalid_argument(\"key can not be nullptr\");\r\n bool retval = false;\r\n READ_TX_BEGIN\r\n Node* node = head->next;\r\n while (true) {\r\n if (node == tail) break;\r\n if (*key == *node->key) {retval = true; break; }\r\n if (*(node->key) < *key) break;\r\n node = node->next;\r\n }\r\n READ_TX_END\r\n return retval;\r\n }\r\n\r\n#else\r\n\r\n /*\r\n * Progress Condition: lock-free\r\n * Adds a node with a key, returns false if the key is already in the set\r\n */\r\n bool add(T* key, const int tid) {\r\n if (key == nullptr) throw std::invalid_argument(\"key can not be nullptr\");\r\n return TM_WRITE_TRANSACTION([this,key] () -> bool {\r\n Node* newNode = TM_ALLOC(key);\r\n Node* prev = head;\r\n Node* node = prev->next;\r\n while (true) {\r\n if (node == tail) break;\r\n if (*key == *node->key) {\r\n TM_FREE(newNode); // If the key was already in the set, free the node that was never used\r\n return false;\r\n }\r\n if (*(node->key) < *key) break;\r\n prev = node;\r\n node = node->next;\r\n }\r\n prev->next = newNode;\r\n newNode->next = node;\r\n return true;\r\n });\r\n }\r\n\r\n\r\n /*\r\n * Progress Condition: lock-free\r\n * Removes a node with an key, returns false if the key is not in the set\r\n */\r\n bool remove(T* key, const int tid) {\r\n if (key == nullptr) throw std::invalid_argument(\"key can not be nullptr\");\r\n return TM_WRITE_TRANSACTION([this,key] () -> bool {\r\n Node* prev = head;\r\n Node* node = prev->next;\r\n while (true) {\r\n if (node == tail) return false;\r\n if (*key == *node->key) {\r\n prev->next = node->next;\r\n TM_FREE(node);\r\n return true;\r\n }\r\n if (*(node->key) < *key) return false;\r\n prev = node;\r\n node = node->next;\r\n }\r\n });\r\n }\r\n\r\n\r\n /*\r\n * Progress Condition: lock-free\r\n * Returns true if it finds a node with a matching key\r\n */\r\n bool contains(T* key, const int tid) {\r\n if (key == nullptr) throw std::invalid_argument(\"key can not be nullptr\");\r\n return TM_READ_TRANSACTION([this,key] () -> bool {\r\n Node* node = head->next;\r\n while (true) {\r\n if (node == tail) return false;\r\n if (*key == *node->key) return true;\r\n if (*(node->key) < *key) return false;\r\n node = node->next;\r\n }\r\n });\r\n }\r\n#endif\r\n\r\n bool addAll(T** keys, int size, const int tid) {\r\n for (int i = 0; i < size; i++) add(keys[i], tid);\r\n }\r\n};\r\n\r\n#endif /* _TM_LINKED_LIST_SET_H_ */\r\n" }, { "path": "datastructures/generic/TMRedBlackBST.hpp", "content": "#ifndef _TM_RED_BLACK_BST_H_\r\n#define _TM_RED_BLACK_BST_H_\r\n\r\n#include \r\n#include \r\n#include \r\n\r\n#include \"../../stms/tm.h\" // This header defines the macros for the STM being compiled\r\n\r\nstatic const int64_t RED = 0;\r\nstatic const int64_t BLACK = 1;\r\n\r\n//http://algs4.cs.princeton.edu/code/edu/princeton/cs/algs4/RedBlackBST.java\r\ntemplate\r\nclass TMRedBlackBST : public TM_BASE_TYPE {\r\n\r\n struct Node : TM_BASE_TYPE {\r\n TM_TYPE key;\r\n TM_TYPE val;\r\n TM_TYPE left {nullptr};\r\n TM_TYPE right {nullptr};\r\n TM_TYPE color; // color of parent link\r\n TM_TYPE size; // subtree count\r\n Node(K* key, V* val, int64_t color, int64_t size) : key{key}, val{val}, color{color}, size{size} {}\r\n };\r\n\r\n TM_TYPE root {nullptr}; // root of the BST\r\n\r\n const unsigned int maxThreads;\r\n\r\n inline void assignAndFreeIfNull(TM_TYPE& z, Node* w) {\r\n Node* tofree = z;\r\n z = w;\r\n if (w == nullptr) TM_FREE(tofree);\r\n }\r\n\r\npublic:\r\n /**\r\n * Initializes an empty symbol table.\r\n */\r\n TMRedBlackBST(unsigned int maxThreads=128) : maxThreads{maxThreads} { }\r\n\r\n /***************************************************************************\r\n * Node helper methods.\r\n ***************************************************************************/\r\n // is node x red; false if x is null ?\r\n bool isRed(Node* x) {\r\n if (x == nullptr) return false;\r\n return x->color == RED;\r\n }\r\n\r\n // number of node in subtree rooted at x; 0 if x is null\r\n int size(Node* x) {\r\n if (x == nullptr) return 0;\r\n return x->size;\r\n }\r\n\r\n\r\n /**\r\n * Returns the number of key-value pairs in this symbol table.\r\n * @return the number of key-value pairs in this symbol table\r\n */\r\n int size() {\r\n return size(root);\r\n }\r\n\r\n /**\r\n * Is this symbol table empty?\r\n * @return {@code true} if this symbol table is empty and {@code false} otherwise\r\n */\r\n bool isEmpty() {\r\n return root == nullptr;\r\n }\r\n\r\n\r\n /***************************************************************************\r\n * Standard BST search->\r\n ***************************************************************************/\r\n\r\n /**\r\n * Returns the value associated with the given key.\r\n * @param key the key\r\n * @return the value associated with the given key if the key is in the symbol table\r\n * and {@code null} if the key is not in the symbol table\r\n * @throws IllegalArgumentException if {@code key} is {@code null}\r\n */\r\n V* get(K* key) {\r\n if (key == nullptr) throw std::invalid_argument(\"item can not be nullptr\");\r\n return get(root, key);\r\n }\r\n\r\n // value associated with the given key in subtree rooted at x; null if no such key\r\n V* get(Node* x, K* key) {\r\n while (x != nullptr) {\r\n if (*key < *x->key) x = x->left;\r\n else if (*x->key < *key) x = x->right;\r\n else return x->val;\r\n }\r\n return nullptr;\r\n }\r\n\r\n /**\r\n * Does this symbol table contain the given key?\r\n * @param key the key\r\n * @return {@code true} if this symbol table contains {@code key} and\r\n * {@code false} otherwise\r\n * @throws IllegalArgumentException if {@code key} is {@code null}\r\n */\r\n bool contains(K* key) {\r\n return get(key) != nullptr;\r\n }\r\n\r\n /***************************************************************************\r\n * Red-black tree insertion.\r\n ***************************************************************************/\r\n\r\n /**\r\n * Inserts the specified key-value pair into the symbol table, overwriting the old\r\n * value with the new value if the symbol table already contains the specified key.\r\n * Deletes the specified key (and its associated value) from this symbol table\r\n * if the specified value is {@code null}.\r\n *\r\n * @param key the key\r\n * @param val the value\r\n * @throws IllegalArgumentException if {@code key} is {@code null}\r\n */\r\n void put(K* key, V* val) {\r\n if (key == nullptr) throw std::invalid_argument(\"item can not be nullptr\");\r\n if (val == nullptr) {\r\n deleteKey(key);\r\n return;\r\n }\r\n root = put(root, key, val);\r\n root->color = BLACK;\r\n // assert check();\r\n }\r\n\r\n // insert the key-value pair in the subtree rooted at h\r\n Node* put(Node* h, K* key, V* val) {\r\n if (h == nullptr) return TM_ALLOC(key, val, RED, 1);\r\n if (*key < *h->key) h->left = put(h->left, key, val);\r\n else if (*h->key < *key) h->right = put(h->right, key, val);\r\n else h->val = val;\r\n // fix-up any right-leaning links\r\n if (isRed(h->right) && !isRed(h->left)) h = rotateLeft(h);\r\n if (isRed(h->left) && isRed(h->left->left)) h = rotateRight(h);\r\n if (isRed(h->left) && isRed(h->right)) flipColors(h);\r\n h->size = size(h->left) + size(h->right) + 1;\r\n\r\n return h;\r\n }\r\n\r\n /***************************************************************************\r\n * Red-black tree deletion.\r\n ***************************************************************************/\r\n\r\n /**\r\n * Removes the smallest key and associated value from the symbol table.\r\n * @throws NoSuchElementException if the symbol table is empty\r\n */\r\n void deleteMin() {\r\n if (isEmpty()) throw std::invalid_argument(\"item can not be nullptr\");\r\n // if both children of root are black, set root to red\r\n if (!isRed(root->left) && !isRed(root->right))\r\n root->color = RED;\r\n assignAndFreeIfNull(root, deleteMin(root));\r\n if (!isEmpty()) root->color = BLACK;\r\n // assert check();\r\n }\r\n\r\n // delete the key-value pair with the minimum key rooted at h\r\n Node* deleteMin(Node* h) {\r\n if (h->left == nullptr)\r\n return nullptr;\r\n if (!isRed(h->left) && !isRed(h->left->left))\r\n h = moveRedLeft(h);\r\n assignAndFreeIfNull(h->left, deleteMin(h->left));\r\n return balance(h);\r\n }\r\n\r\n\r\n /**\r\n * Removes the largest key and associated value from the symbol table.\r\n * @throws NoSuchElementException if the symbol table is empty\r\n */\r\n void deleteMax() {\r\n if (isEmpty()) throw std::invalid_argument(\"item can not be nullptr\");\r\n\r\n // if both children of root are black, set root to red\r\n if (!isRed(root->left) && !isRed(root->right))\r\n root->color = RED;\r\n\r\n root = deleteMax(root);\r\n if (!isEmpty()) root->color = BLACK;\r\n // assert check();\r\n }\r\n\r\n // delete the key-value pair with the maximum key rooted at h\r\n Node* deleteMax(Node* h) {\r\n if (isRed(h->left))\r\n h = rotateRight(h);\r\n\r\n if (h->right == nullptr)\r\n return nullptr;\r\n\r\n if (!isRed(h->right) && !isRed(h->right->left))\r\n h = moveRedRight(h);\r\n\r\n h->right = deleteMax(h->right);\r\n\r\n return balance(h);\r\n }\r\n\r\n /**\r\n * Removes the specified key and its associated value from this symbol table\r\n * (if the key is in this symbol table).\r\n *\r\n * @param key the key\r\n * @throws IllegalArgumentException if {@code key} is {@code null}\r\n */\r\n void deleteKey(K* key) {\r\n if (key == nullptr) throw std::invalid_argument(\"item can not be nullptr\");\r\n if (!contains(key)) return;\r\n // if both children of root are black, set root to red\r\n if (!isRed(root->left) && !isRed(root->right))\r\n root->color = RED;\r\n assignAndFreeIfNull(root, deleteKey(root, key));\r\n if (!isEmpty()) root->color = BLACK;\r\n // assert check();\r\n }\r\n\r\n // delete the key-value pair with the given key rooted at h\r\n Node* deleteKey(Node* h, K* key) {\r\n // assert get(h, key) != null;\r\n if (*key < *h->key) {\r\n if (!isRed(h->left) && !isRed(h->left->left)) {\r\n h = moveRedLeft(h);\r\n }\r\n assignAndFreeIfNull(h->left, deleteKey(h->left, key));\r\n } else {\r\n if (isRed(h->left)) {\r\n h = rotateRight(h);\r\n }\r\n if (*key == *h->key && (h->right == nullptr)) {\r\n return nullptr;\r\n }\r\n if (!isRed(h->right) && !isRed(h->right->left)) {\r\n h = moveRedRight(h);\r\n }\r\n if (*key == *h->key) {\r\n Node* x = min(h->right);\r\n h->key = x->key;\r\n h->val = x->val;\r\n // h->val = get(h->right, min(h->right).key);\r\n // h->key = min(h->right).key;\r\n assignAndFreeIfNull(h->right, deleteMin(h->right));\r\n } else {\r\n assignAndFreeIfNull(h->right, deleteKey(h->right, key));\r\n }\r\n }\r\n return balance(h);\r\n }\r\n\r\n /***************************************************************************\r\n * Red-black tree helper functions.\r\n ***************************************************************************/\r\n\r\n // make a left-leaning link lean to the right\r\n Node* rotateRight(Node* h) {\r\n // assert (h != null) && isRed(h->left);\r\n Node* x = h->left;\r\n h->left = x->right;\r\n x->right = h;\r\n x->color = x->right->color;\r\n x->right->color = RED;\r\n x->size = h->size;\r\n h->size = size(h->left) + size(h->right) + 1;\r\n return x;\r\n }\r\n\r\n // make a right-leaning link lean to the left\r\n Node* rotateLeft(Node* h) {\r\n // assert (h != null) && isRed(h->right);\r\n Node* x = h->right;\r\n h->right = x->left;\r\n x->left = h;\r\n x->color = x->left->color;\r\n x->left->color = RED;\r\n x->size = h->size;\r\n h->size = size(h->left) + size(h->right) + 1;\r\n return x;\r\n }\r\n\r\n // flip the colors of a node and its two children\r\n void flipColors(Node* h) {\r\n // h must have opposite color of its two children\r\n // assert (h != null) && (h->left != null) && (h->right != null);\r\n // assert (!isRed(h) && isRed(h->left) && isRed(h->right))\r\n // || (isRed(h) && !isRed(h->left) && !isRed(h->right));\r\n h->color = !h->color;\r\n h->left->color = !h->left->color;\r\n h->right->color = !h->right->color;\r\n }\r\n\r\n // Assuming that h is red and both h->left and h->left.left\r\n // are black, make h->left or one of its children red.\r\n Node* moveRedLeft(Node* h) {\r\n // assert (h != null);\r\n // assert isRed(h) && !isRed(h->left) && !isRed(h->left.left);\r\n\r\n flipColors(h);\r\n if (isRed(h->right->left)) {\r\n h->right = rotateRight(h->right);\r\n h = rotateLeft(h);\r\n flipColors(h);\r\n }\r\n return h;\r\n }\r\n\r\n // Assuming that h is red and both h->right and h->right.left\r\n // are black, make h->right or one of its children red.\r\n Node* moveRedRight(Node* h) {\r\n // assert (h != null);\r\n // assert isRed(h) && !isRed(h->right) && !isRed(h->right.left);\r\n flipColors(h);\r\n if (isRed(h->left->left)) {\r\n h = rotateRight(h);\r\n flipColors(h);\r\n }\r\n return h;\r\n }\r\n\r\n // restore red-black tree invariant\r\n Node* balance(Node* h) {\r\n // assert (h != null);\r\n\r\n if (isRed(h->right)) h = rotateLeft(h);\r\n if (isRed(h->left) && isRed(h->left->left)) h = rotateRight(h);\r\n if (isRed(h->left) && isRed(h->right)) flipColors(h);\r\n\r\n h->size = size(h->left) + size(h->right) + 1;\r\n return h;\r\n }\r\n\r\n\r\n /***************************************************************************\r\n * Utility functions.\r\n ***************************************************************************/\r\n\r\n /**\r\n * Returns the height of the BST (for debugging).\r\n * @return the height of the BST (a 1-node tree has height 0)\r\n */\r\n int height() {\r\n return height(root);\r\n }\r\n int height(Node* x) {\r\n if (x == nullptr) return -1;\r\n return 1 + std::max(height(x->left), height(x->right));\r\n }\r\n\r\n /***************************************************************************\r\n * Ordered symbol table methods.\r\n ***************************************************************************/\r\n\r\n /**\r\n * Returns the smallest key in the symbol table.\r\n * @return the smallest key in the symbol table\r\n * @throws NoSuchElementException if the symbol table is empty\r\n */\r\n K* min() {\r\n if (isEmpty()) throw std::invalid_argument(\"item can not be nullptr\");\r\n return min(root).key;\r\n }\r\n\r\n // the smallest key in subtree rooted at x; null if no such key\r\n Node* min(Node* x) {\r\n // assert x != null;\r\n if (x->left == nullptr) return x;\r\n else return min(x->left);\r\n }\r\n\r\n /**\r\n * Returns the largest key in the symbol table.\r\n * @return the largest key in the symbol table\r\n * @throws NoSuchElementException if the symbol table is empty\r\n */\r\n K* max() {\r\n if (isEmpty()) throw std::invalid_argument(\"item can not be nullptr\");\r\n return max(root).key;\r\n }\r\n\r\n // the largest key in the subtree rooted at x; null if no such key\r\n Node* max(Node* x) {\r\n // assert x != null;\r\n if (x->right == nullptr) return x;\r\n else return max(x->right);\r\n }\r\n\r\n\r\n /**\r\n * Returns the largest key in the symbol table less than or equal to {@code key}.\r\n * @param key the key\r\n * @return the largest key in the symbol table less than or equal to {@code key}\r\n * @throws NoSuchElementException if there is no such key\r\n * @throws IllegalArgumentException if {@code key} is {@code null}\r\n */\r\n K* floor(K* key) {\r\n if (key == nullptr) throw std::invalid_argument(\"item can not be nullptr\");\r\n if (isEmpty()) throw std::invalid_argument(\"item can not be nullptr\");\r\n Node* x = floor(root, key);\r\n if (x == nullptr) return nullptr;\r\n else return x->key;\r\n }\r\n\r\n // the largest key in the subtree rooted at x less than or equal to the given key\r\n Node* floor(Node* x, K* key) {\r\n if (x == nullptr) return nullptr;\r\n if (*key == *x->key) return x;\r\n if (*key < *x->key) return floor(x->left, key);\r\n Node* t = floor(x->right, key);\r\n if (t != nullptr) return t;\r\n else return x;\r\n }\r\n\r\n /**\r\n * Returns the smallest key in the symbol table greater than or equal to {@code key}.\r\n * @param key the key\r\n * @return the smallest key in the symbol table greater than or equal to {@code key}\r\n * @throws NoSuchElementException if there is no such key\r\n * @throws IllegalArgumentException if {@code key} is {@code null}\r\n */\r\n K* ceiling(K* key) {\r\n if (key == nullptr) throw std::invalid_argument(\"item can not be nullptr\");\r\n if (isEmpty()) throw std::invalid_argument(\"item can not be nullptr\");\r\n Node* x = ceiling(root, key);\r\n if (x == nullptr) return nullptr;\r\n else return x->key;\r\n }\r\n\r\n // the smallest key in the subtree rooted at x greater than or equal to the given key\r\n Node* ceiling(Node* x, K* key) {\r\n if (x == nullptr) return nullptr;\r\n if (*key == *x->key) return x;\r\n if (*x->key < *key) return ceiling(x->right, key);\r\n Node* t = ceiling(x->left, key);\r\n if (t != nullptr) return t;\r\n else return x;\r\n }\r\n\r\n /**\r\n * Return the kth smallest key in the symbol table.\r\n * @param k the order statistic\r\n * @return the {@code k}th smallest key in the symbol table\r\n * @throws IllegalArgumentException unless {@code k} is between 0 and\r\n * n1\r\n */\r\n K* select(int k) {\r\n if (k < 0 || k >= size()) {\r\n throw std::invalid_argument(\"item can not be nullptr\");\r\n }\r\n Node x = select(root, k);\r\n return x->key;\r\n }\r\n\r\n // the key of rank k in the subtree rooted at x\r\n Node* select(Node* x, int k) {\r\n // assert x != null;\r\n // assert k >= 0 && k < size(x);\r\n int t = size(x->left);\r\n if (t > k) return select(x->left, k);\r\n else if (t < k) return select(x->right, k-t-1);\r\n else return x;\r\n }\r\n\r\n /**\r\n * Return the number of keys in the symbol table strictly less than {@code key}.\r\n * @param key the key\r\n * @return the number of keys in the symbol table strictly less than {@code key}\r\n * @throws IllegalArgumentException if {@code key} is {@code null}\r\n */\r\n int rank(K* key) {\r\n if (key == nullptr) throw std::invalid_argument(\"item can not be nullptr\");\r\n return rank(key, root);\r\n }\r\n\r\n // number of keys less than key in the subtree rooted at x\r\n int rank(K* key, Node* x) {\r\n if (x == nullptr) return 0;\r\n if (*key < *x->key) return rank(key, x->left);\r\n else if (*x->key < *key) return 1 + size(x->left) + rank(key, x->right);\r\n else return size(x->left);\r\n }\r\n\r\n /***************************************************************************\r\n * Range count and range search->\r\n ***************************************************************************/\r\n\r\n\r\n /**\r\n * Returns the number of keys in the symbol table in the given range.\r\n *\r\n * @param lo minimum endpoint\r\n * @param hi maximum endpoint\r\n * @return the number of keys in the sybol table between {@code lo}\r\n * (inclusive) and {@code hi} (inclusive)\r\n * @throws IllegalArgumentException if either {@code lo} or {@code hi}\r\n * is {@code null}\r\n */\r\n int size(K* lo, K* hi) {\r\n if (lo == nullptr) throw std::invalid_argument(\"item can not be nullptr\");\r\n if (hi == nullptr) throw std::invalid_argument(\"item can not be nullptr\");\r\n\r\n if (*hi < *lo) return 0;\r\n if (contains(hi)) return rank(hi) - rank(lo) + 1;\r\n else return rank(hi) - rank(lo);\r\n }\r\n\r\n\r\n /***************************************************************************\r\n * Check integrity of red-black tree data structure.\r\n ***************************************************************************/\r\n bool check() {\r\n if (!isBST()) std::cout << \"Not in symmetric order\\n\";\r\n if (!isSizeConsistent()) std::cout << \"Subtree counts not consistent\\n\";\r\n //if (!isRankConsistent()) std::cout << \"Ranks not consistent\\n\";\r\n if (!is23()) std::cout << \"Not a 2-3 tree\\n\";\r\n if (!isBalanced()) std::cout << \"Not balanced\\n\";\r\n return isBST() && isSizeConsistent() && is23() && isBalanced();\r\n }\r\n\r\n // does this binary tree satisfy symmetric order?\r\n // Note: this test also ensures that data structure is a binary tree since order is strict\r\n bool isBST() {\r\n return isBST(root, nullptr, nullptr);\r\n }\r\n\r\n // is the tree rooted at x a BST with all keys strictly between min and max\r\n // (if min or max is null, treat as empty constraint)\r\n // Credit: Bob Dondero's elegant solution\r\n bool isBST(Node* x, K* min, K* max) {\r\n if (x == nullptr) return true;\r\n // TODO: port these two lines\r\n //if (min != nullptr && x->key.compareTo(min) <= 0) return false;\r\n //if (max != nullptr && x->key.compareTo(max) >= 0) return false;\r\n return isBST(x->left, min, x->key) && isBST(x->right, x->key, max);\r\n }\r\n\r\n // are the size fields correct?\r\n bool isSizeConsistent() { return isSizeConsistent(root); }\r\n bool isSizeConsistent(Node* x) {\r\n if (x == nullptr) return true;\r\n if (x->size != size(x->left) + size(x->right) + 1) return false;\r\n return isSizeConsistent(x->left) && isSizeConsistent(x->right);\r\n }\r\n\r\n /*\r\n // check that ranks are consistent\r\n bool isRankConsistent() {\r\n for (int i = 0; i < size(); i++)\r\n if (i != rank(select(i))) return false;\r\n for (K* key : keys())\r\n if (key.compareTo(select(rank(key))) != 0) return false;\r\n return true;\r\n }\r\n */\r\n\r\n // Does the tree have no red right links, and at most one (left)\r\n // red links in a row on any path?\r\n bool is23() { return is23(root); }\r\n bool is23(Node* x) {\r\n if (x == nullptr) return true;\r\n if (isRed(x->right)) return false;\r\n if (x != root && isRed(x) && isRed(x->left))\r\n return false;\r\n return is23(x->left) && is23(x->right);\r\n }\r\n\r\n // do all paths from root to leaf have same number of black edges?\r\n bool isBalanced() {\r\n int black = 0; // number of black links on path from root to min\r\n Node x = root;\r\n while (x != nullptr) {\r\n if (!isRed(x)) black++;\r\n x = x->left;\r\n }\r\n return isBalanced(root, black);\r\n }\r\n\r\n // does every path from the root to a leaf have the given number of black links?\r\n bool isBalanced(Node* x, int black) {\r\n if (x == nullptr) return black == 0;\r\n if (!isRed(x)) black--;\r\n return isBalanced(x->left, black) && isBalanced(x->right, black);\r\n }\r\n\r\n\r\n\r\n // Set methods\r\n bool add(K* key, const int tid) {\r\n return TM_WRITE_TRANSACTION([this,key] () -> bool {\r\n if (contains(key)) return false;\r\n put(key,key);\r\n return true;\r\n });\r\n }\r\n\r\n bool remove(K* key, const int tid) {\r\n return TM_WRITE_TRANSACTION([this,key] () -> bool {\r\n if (!contains(key)) return false;\r\n deleteKey(key);\r\n return true;\r\n });\r\n }\r\n\r\n inline bool contains(K* key, const int tid) {\r\n return TM_READ_TRANSACTION([this,key] () -> bool {\r\n return contains(key);\r\n });\r\n }\r\n\r\n // This is not fully transactionally but it's ok because we use it only on initialization.\r\n // We could make it fully transactionally, but we would have to increase the size of allocation/store logs.\r\n bool addAll(K** keys, int size, const int tid) {\r\n for (int i = 0; i < size; i++) add(keys[i], tid);\r\n }\r\n\r\n std::string className() { return TM_NAME() + \"-RedBlackBST\"; }\r\n\r\n};\r\n\r\n#endif // _TM_RED_BLACK_BST_H_\r\n" }, { "path": "datastructures/hashmaps/CRWWPSTMResizableHashSet.hpp", "content": "/*\n * Copyright 2017-2018\n * Andreia Correia \n * Pedro Ramalhete \n * Pascal Felber \n * Nachshon Cohen \n *\n * This work is published under the MIT license. See LICENSE.txt\n */\n#ifndef _CRWWP_STM_RESIZABLE_HASH_MAP_H_\n#define _CRWWP_STM_RESIZABLE_HASH_MAP_H_\n\n#include \n\n#include \"stms/CRWWPSTM.hpp\"\n\n/**\n *

A Resizable Hash Map for usage with STMs

\n * TODO\n *\n */\ntemplate\nclass CRWWPSTMResizableHashSet {\n\nprivate:\n struct Node : public crwwpstm::tmbase {\n crwwpstm::tmtype key;\n crwwpstm::tmtype next {nullptr};\n Node(const K& k) : key{k} { } // Copy constructor for k\n };\n\n crwwpstm::tmtype capacity;\n crwwpstm::tmtype sizeHM = 0;\n static constexpr double loadFactor = 0.75;\n crwwpstm::tmtype*> buckets; // An array of pointers to Nodes\n\n\npublic:\n CRWWPSTMResizableHashSet(int maxThreads=0, int capacity=4) : capacity{capacity} {\n crwwpstm::updateTx([&] () {\n buckets = (crwwpstm::tmtype*)crwwpstm::tmMalloc(capacity*sizeof(crwwpstm::tmtype));\n for (int i = 0; i < capacity; i++) buckets[i] = nullptr;\n });\n }\n\n\n ~CRWWPSTMResizableHashSet() {\n crwwpstm::updateTx([&] () {\n for(int i = 0; i < capacity; i++){\n Node* node = buckets[i];\n while (node != nullptr) {\n Node* next = node->next;\n crwwpstm::tmDelete(node);\n node = next;\n }\n }\n crwwpstm::tmFree(buckets.load());\n });\n }\n\n\n static std::string className() { return crwwpstm::CRWWPSTM::className() + \"-HashMap\"; }\n\n\n void rebuild() {\n int newcapacity = 2*capacity;\n crwwpstm::tmtype* newbuckets = (crwwpstm::tmtype*)crwwpstm::tmMalloc(newcapacity*sizeof(crwwpstm::tmtype));\n for (int i = 0; i < newcapacity; i++) newbuckets[i] = nullptr;\n for (int i = 0; i < capacity; i++) {\n Node* node = buckets[i];\n while(node!=nullptr){\n Node* next = node->next;\n auto h = std::hash{}(node->key) % newcapacity;\n node->next = newbuckets[h];\n newbuckets[h] = node;\n node = next;\n }\n }\n crwwpstm::tmFree(buckets.load());\n buckets = newbuckets;\n capacity = newcapacity;\n }\n\n\n /*\n * Adds a node with a key if the key is not present, otherwise replaces the value.\n * If saveOldValue is set, it will set 'oldValue' to the previous value, iff there was already a mapping.\n *\n * Returns true if there was no mapping for the key, false if there was already a value and it was replaced.\n */\n bool innerPut(const K& key) {\n if (sizeHM > capacity*loadFactor) rebuild();\n auto h = std::hash{}(key) % capacity;\n Node* node = buckets[h];\n Node* prev = node;\n while (true) {\n if (node == nullptr) {\n Node* newnode = crwwpstm::tmNew(key);\n if (node == prev) {\n buckets[h] = newnode;\n } else {\n prev->next = newnode;\n }\n sizeHM++;\n return true; // New insertion\n }\n if (key == node->key) return false;\n prev = node;\n node = node->next;\n }\n }\n\n\n /*\n * Removes a key and its mapping.\n * Saves the value in 'oldvalue' if 'saveOldValue' is set.\n *\n * Returns returns true if a matching key was found\n */\n bool innerRemove(const K& key) {\n auto h = std::hash{}(key) % capacity;\n Node* node = buckets[h];\n Node* prev = node;\n while (true) {\n if (node == nullptr) return false;\n if (key == node->key) {\n if (node == prev) {\n buckets[h] = node->next;\n } else {\n prev->next = node->next;\n }\n sizeHM--;\n crwwpstm::tmDelete(node);\n return true;\n }\n prev = node;\n node = node->next;\n }\n }\n\n\n /*\n * Returns true if key is present. Saves a copy of 'value' in 'oldValue' if 'saveOldValue' is set.\n */\n bool innerGet(const K& key) {\n auto h = std::hash{}(key) % capacity;\n Node* node = buckets[h];\n while (true) {\n if (node == nullptr) return false;\n if (key == node->key) return true;\n node = node->next;\n }\n }\n\n\n //\n // Set methods for running the usual tests and benchmarks\n //\n\n // Inserts a key only if it's not already present\n bool add(K key, const int tid=0) {\n return crwwpstm::updateTx([&] () {\n return innerPut(key);\n });\n }\n\n // Returns true only if the key was present\n bool remove(K key, const int tid=0) {\n return crwwpstm::updateTx([&] () {\n return innerRemove(key);\n });\n }\n\n bool contains(K key, const int tid=0) {\n return crwwpstm::readTx([&] () {\n return innerGet(key);\n });\n }\n\n // Used only for benchmarks\n void addAll(K** keys, const int size, const int tid=0) {\n for (int i = 0; i < size; i++) add(*keys[i]);\n }\n};\n\n#endif /* _CRWWP_STM_RESIZABLE_HASH_MAP_H_ */\n" }, { "path": "datastructures/hashmaps/ESTMResizableHashSet.hpp", "content": "/*\n * Copyright 2017-2018\n * Andreia Correia \n * Pedro Ramalhete \n * Pascal Felber \n * Nachshon Cohen \n *\n * This work is published under the MIT license. See LICENSE.txt\n */\n#ifndef _ESTM_RESIZABLE_HASH_MAP_H_\n#define _ESTM_RESIZABLE_HASH_MAP_H_\n\n#include \n\n#include \"stms/ESTM.hpp\"\n\n/**\n *

A Resizable Hash Map for usage with STMs

\n * TODO\n *\n */\ntemplate\nclass ESTMResizableHashSet {\n\nprivate:\n struct Node : public estm::tmbase {\n estm::tmtype key;\n estm::tmtype next {nullptr};\n Node(const K& k) : key{k} { } // Copy constructor for k\n };\n\n estm::tmtype capacity;\n estm::tmtype sizeHM = 0;\n static constexpr double loadFactor = 0.75;\n estm::tmtype*> buckets; // An array of pointers to Nodes\n\n\npublic:\n ESTMResizableHashSet(int maxThreads=0, uint64_t capacity=4) : capacity{capacity} {\n estm::updateTx([&] () {\n buckets = (estm::tmtype*)estm::tmMalloc(capacity*sizeof(estm::tmtype));\n for (int i = 0; i < capacity; i++) buckets[i] = nullptr;\n });\n }\n\n\n ~ESTMResizableHashSet() {\n estm::updateTx([&] () {\n for(int i = 0; i < capacity; i++){\n Node* node = buckets[i];\n while (node != nullptr) {\n Node* next = node->next;\n estm::tmDelete(node);\n node = next;\n }\n }\n estm::tmFree(buckets.load());\n });\n }\n\n\n static std::string className() { return estm::ESTM::className() + \"-HashMap\"; }\n\n\n void rebuild() {\n uint64_t newcapacity = 2*capacity;\n estm::tmtype* newbuckets = (estm::tmtype*)estm::tmMalloc(newcapacity*sizeof(estm::tmtype));\n for (int i = 0; i < newcapacity; i++) newbuckets[i] = nullptr;\n for (int i = 0; i < capacity; i++) {\n Node* node = buckets[i];\n while (node!=nullptr) {\n Node* next = node->next;\n auto h = std::hash{}(node->key) % newcapacity;\n node->next = newbuckets[h];\n newbuckets[h] = node;\n node = next;\n }\n }\n estm::tmFree(buckets);\n buckets = newbuckets;\n capacity = newcapacity;\n }\n\n\n /*\n * Adds a node with a key if the key is not present, otherwise replaces the value.\n * If saveOldValue is set, it will set 'oldValue' to the previous value, iff there was already a mapping.\n *\n * Returns true if there was no mapping for the key, false if there was already a value and it was replaced.\n */\n bool innerPut(const K& key) {\n if (sizeHM.load() > capacity.load()*loadFactor) rebuild();\n auto h = std::hash{}(key) % capacity;\n Node* node = buckets[h];\n Node* prev = node;\n while (true) {\n if (node == nullptr) {\n Node* newnode = estm::tmNew(key);\n if (node == prev) {\n buckets[h] = newnode;\n } else {\n prev->next = newnode;\n }\n sizeHM++;\n return true; // New insertion\n }\n if (key == node->key) return false;\n prev = node;\n node = node->next;\n }\n }\n\n\n /*\n * Removes a key and its mapping.\n * Saves the value in 'oldvalue' if 'saveOldValue' is set.\n *\n * Returns returns true if a matching key was found\n */\n bool innerRemove(const K& key) {\n auto h = std::hash{}(key) % capacity;\n Node* node = buckets[h];\n Node* prev = node;\n while (true) {\n if (node == nullptr) return false;\n if (key == node->key) {\n if (node == prev) {\n buckets[h] = node->next;\n } else {\n prev->next = node->next;\n }\n sizeHM--;\n estm::tmDelete(node);\n return true;\n }\n prev = node;\n node = node->next;\n }\n }\n\n\n /*\n * Returns true if key is present. Saves a copy of 'value' in 'oldValue' if 'saveOldValue' is set.\n */\n bool innerGet(const K& key) {\n auto h = std::hash{}(key) % capacity;\n Node* node = buckets[h];\n while (true) {\n if (node == nullptr) return false;\n if (key == node->key) return true;\n node = node->next;\n }\n }\n\n\n //\n // Set methods for running the usual tests and benchmarks\n //\n\n // Inserts a key only if it's not already present\n bool add(K key, const int tid=0) {\n return estm::updateTx([&] () {\n return innerPut(key);\n });\n }\n\n // Returns true only if the key was present\n bool remove(K key, const int tid=0) {\n return estm::updateTx([&] () {\n return innerRemove(key);\n });\n }\n\n bool contains(K key, const int tid=0) {\n return estm::readTx([&] () {\n return innerGet(key);\n });\n }\n\n // Used only for benchmarks\n void addAll(K** keys, const int size, const int tid=0) {\n for (int i = 0; i < size; i++) add(*keys[i]);\n }\n};\n\n#endif /* _ESTM_RESIZABLE_HASH_MAP_H_ */\n" }, { "path": "datastructures/hashmaps/OFLFResizableHashSet.hpp", "content": "/*\n * Copyright 2017-2018\n * Andreia Correia \n * Pedro Ramalhete \n * Pascal Felber \n * Nachshon Cohen \n *\n * This work is published under the MIT license. See LICENSE.txt\n */\n#ifndef _OF_LF_RESIZABLE_HASH_MAP_H_\n#define _OF_LF_RESIZABLE_HASH_MAP_H_\n\n#include \n\n#include \"stms/OneFileLF.hpp\"\n\n/**\n *

A Resizable Hash Map for usage with STMs

\n * TODO\n *\n */\ntemplate\nclass OFLFResizableHashSet {\n\nprivate:\n struct Node : public oflf::tmbase {\n oflf::tmtype key;\n oflf::tmtype next {nullptr};\n Node(const K& k) : key{k} { } // Copy constructor for k\n };\n\n oflf::tmtype capacity;\n oflf::tmtype sizeHM = 0;\n static constexpr double loadFactor = 0.75;\n oflf::tmtype*> buckets; // An array of pointers to Nodes\n\n\npublic:\n OFLFResizableHashSet(int maxThreads=0, uint64_t capacity=4) : capacity{capacity} {\n oflf::updateTx([&] () {\n buckets = (oflf::tmtype*)oflf::tmMalloc(capacity*sizeof(oflf::tmtype));\n for (int i = 0; i < capacity; i++) buckets[i] = nullptr;\n });\n }\n\n\n ~OFLFResizableHashSet() {\n oflf::updateTx([&] () {\n for (int i = 0; i < capacity; i++){\n Node* node = buckets[i];\n while (node != nullptr) {\n Node* next = node->next;\n oflf::tmDelete(node);\n node = next;\n }\n }\n oflf::tmFree(buckets.pload());\n });\n }\n\n\n static std::string className() { return oflf::OneFileLF::className() + \"-HashMap\"; }\n\n\n void rebuild() {\n uint64_t newcapacity = 2*capacity;\n oflf::tmtype* newbuckets = (oflf::tmtype*)oflf::tmMalloc(newcapacity*sizeof(oflf::tmtype));\n for (int i = 0; i < newcapacity; i++) newbuckets[i] = nullptr;\n for (int i = 0; i < capacity; i++) {\n Node* node = buckets[i];\n while (node!=nullptr) {\n Node* next = node->next;\n auto h = std::hash{}(node->key) % newcapacity;\n node->next = newbuckets[h];\n newbuckets[h] = node;\n node = next;\n }\n }\n oflf::tmFree(buckets.pload());\n buckets = newbuckets;\n capacity = newcapacity;\n }\n\n\n /*\n * Adds a node with a key if the key is not present, otherwise replaces the value.\n * If saveOldValue is set, it will set 'oldValue' to the previous value, iff there was already a mapping.\n *\n * Returns true if there was no mapping for the key, false if there was already a value and it was replaced.\n */\n bool innerPut(const K& key) {\n if (sizeHM.pload() > capacity.pload()*loadFactor) rebuild();\n auto h = std::hash{}(key) % capacity;\n Node* node = buckets[h];\n Node* prev = node;\n while (true) {\n if (node == nullptr) {\n Node* newnode = oflf::tmNew(key);\n if (node == prev) {\n buckets[h] = newnode;\n } else {\n prev->next = newnode;\n }\n sizeHM++;\n return true; // New insertion\n }\n if (key == node->key) return false;\n prev = node;\n node = node->next;\n }\n }\n\n\n /*\n * Removes a key and its mapping.\n * Saves the value in 'oldvalue' if 'saveOldValue' is set.\n *\n * Returns returns true if a matching key was found\n */\n bool innerRemove(const K& key) {\n auto h = std::hash{}(key) % capacity;\n Node* node = buckets[h];\n Node* prev = node;\n while (true) {\n if (node == nullptr) return false;\n if (key == node->key) {\n if (node == prev) {\n buckets[h] = node->next;\n } else {\n prev->next = node->next;\n }\n sizeHM--;\n oflf::tmDelete(node);\n return true;\n }\n prev = node;\n node = node->next;\n }\n }\n\n\n /*\n * Returns true if key is present. Saves a copy of 'value' in 'oldValue' if 'saveOldValue' is set.\n */\n bool innerGet(const K& key) {\n auto h = std::hash{}(key) % capacity;\n Node* node = buckets[h];\n while (true) {\n if (node == nullptr) return false;\n if (key == node->key) return true;\n node = node->next;\n }\n }\n\n\n //\n // Set methods for running the usual tests and benchmarks\n //\n\n // Inserts a key only if it's not already present\n bool add(K key, const int tid=0) {\n return oflf::updateTx([&] () {\n return innerPut(key);\n });\n }\n\n // Returns true only if the key was present\n bool remove(K key, const int tid=0) {\n return oflf::updateTx([&] () {\n return innerRemove(key);\n });\n }\n\n bool contains(K key, const int tid=0) {\n return oflf::readTx([&] () {\n return innerGet(key);\n });\n }\n\n // Used only for benchmarks\n void addAll(K** keys, const int size, const int tid=0) {\n for (int i = 0; i < size; i++) add(*keys[i]);\n }\n};\n\n#endif /* _OF_LF_RESIZABLE_HASH_MAP_H_ */\n" }, { "path": "datastructures/hashmaps/OFWFResizableHashSet.hpp", "content": "/*\n * Copyright 2017-2018\n * Andreia Correia \n * Pedro Ramalhete \n * Pascal Felber \n * Nachshon Cohen \n *\n * This work is published under the MIT license. See LICENSE.txt\n */\n#ifndef _OF_WF_RESIZABLE_HASH_MAP_H_\n#define _OF_WF_RESIZABLE_HASH_MAP_H_\n\n#include \n\n#include \"stms/OneFileWF.hpp\"\n\n/**\n *

A Resizable Hash Map for usage with STMs

\n * TODO\n *\n */\ntemplate\nclass OFWFResizableHashSet {\n\nprivate:\n struct Node : public ofwf::tmbase {\n ofwf::tmtype key;\n ofwf::tmtype next {nullptr};\n Node(const K& k) : key{k} { } // Copy constructor for k\n };\n\n ofwf::tmtype capacity;\n ofwf::tmtype sizeHM = 0;\n static constexpr double loadFactor = 0.75;\n ofwf::tmtype*> buckets; // An array of pointers to Nodes\n\n\npublic:\n OFWFResizableHashSet(int maxThreads=0, uint64_t capacity=4) : capacity{capacity} {\n ofwf::updateTx([&] () {\n buckets = (ofwf::tmtype*)ofwf::tmMalloc(capacity*sizeof(ofwf::tmtype));\n for (int i = 0; i < capacity; i++) buckets[i] = nullptr;\n });\n }\n\n\n ~OFWFResizableHashSet() {\n ofwf::updateTx([=] () {\n for (int i = 0; i < capacity; i++){\n Node* node = buckets[i];\n while (node != nullptr) {\n Node* next = node->next;\n ofwf::tmDelete(node);\n node = next;\n }\n }\n ofwf::tmFree(buckets.pload());\n });\n }\n\n\n static std::string className() { return ofwf::OneFileWF::className() + \"-HashMap\"; }\n\n\n void rebuild() {\n uint64_t newcapacity = 2*capacity;\n ofwf::tmtype* newbuckets = (ofwf::tmtype*)ofwf::tmMalloc(newcapacity*sizeof(ofwf::tmtype));\n for (int i = 0; i < newcapacity; i++) newbuckets[i] = nullptr;\n for (int i = 0; i < capacity; i++) {\n Node* node = buckets[i];\n while (node!=nullptr) {\n Node* next = node->next;\n auto h = std::hash{}(node->key) % newcapacity;\n node->next = newbuckets[h];\n newbuckets[h] = node;\n node = next;\n }\n }\n ofwf::tmFree(buckets.pload());\n buckets = newbuckets;\n capacity = newcapacity;\n }\n\n\n /*\n * Adds a node with a key if the key is not present, otherwise replaces the value.\n * If saveOldValue is set, it will set 'oldValue' to the previous value, iff there was already a mapping.\n *\n * Returns true if there was no mapping for the key, false if there was already a value and it was replaced.\n */\n bool innerPut(const K& key) {\n if (sizeHM.pload() > capacity.pload()*loadFactor) rebuild();\n auto h = std::hash{}(key) % capacity;\n Node* node = buckets[h];\n Node* prev = node;\n while (true) {\n if (node == nullptr) {\n Node* newnode = ofwf::tmNew(key);\n if (node == prev) {\n buckets[h] = newnode;\n } else {\n prev->next = newnode;\n }\n sizeHM++;\n return true; // New insertion\n }\n if (key == node->key) return false;\n prev = node;\n node = node->next;\n }\n }\n\n\n /*\n * Removes a key and its mapping.\n * Saves the value in 'oldvalue' if 'saveOldValue' is set.\n *\n * Returns returns true if a matching key was found\n */\n bool innerRemove(const K& key) {\n auto h = std::hash{}(key) % capacity;\n Node* node = buckets[h];\n Node* prev = node;\n while (true) {\n if (node == nullptr) return false;\n if (key == node->key) {\n if (node == prev) {\n buckets[h] = node->next;\n } else {\n prev->next = node->next;\n }\n sizeHM--;\n ofwf::tmDelete(node);\n return true;\n }\n prev = node;\n node = node->next;\n }\n }\n\n\n /*\n * Returns true if key is present. Saves a copy of 'value' in 'oldValue' if 'saveOldValue' is set.\n */\n bool innerGet(const K& key) {\n auto h = std::hash{}(key) % capacity;\n Node* node = buckets[h];\n while (true) {\n if (node == nullptr) return false;\n if (key == node->key) return true;\n node = node->next;\n }\n }\n\n\n //\n // Set methods for running the usual tests and benchmarks\n //\n\n // Inserts a key only if it's not already present\n bool add(K key, const int tid=0) {\n return ofwf::updateTx([=] () {\n return innerPut(key);\n });\n }\n\n // Returns true only if the key was present\n bool remove(K key, const int tid=0) {\n return ofwf::updateTx([=] () {\n return innerRemove(key);\n });\n }\n\n bool contains(K key, const int tid=0) {\n return ofwf::readTx([=] () {\n return innerGet(key);\n });\n }\n\n // Used only for benchmarks\n void addAll(K** keys, const int size, const int tid=0) {\n for (int i = 0; i < size; i++) add(*keys[i]);\n }\n};\n\n#endif /* _OF_WF_RESIZABLE_HASH_MAP_H_ */\n" }, { "path": "datastructures/hashmaps/TinySTMResizableHashSet.hpp", "content": "/*\n * Copyright 2017-2018\n * Andreia Correia \n * Pedro Ramalhete \n * Pascal Felber \n * Nachshon Cohen \n *\n * This work is published under the MIT license. See LICENSE.txt\n */\n#ifndef _TINY_STM_RESIZABLE_HASH_MAP_H_\n#define _TINY_STM_RESIZABLE_HASH_MAP_H_\n\n#include \n\n#include \"stms/TinySTM.hpp\"\n\n/**\n *

A Resizable Hash Map for usage with STMs

\n * TODO\n *\n */\ntemplate\nclass TinySTMResizableHashSet {\n\nprivate:\n struct Node : public tinystm::tmbase {\n tinystm::tmtype key;\n tinystm::tmtype next {nullptr};\n Node(const K& k) : key{k} { } // Copy constructor for k\n };\n\n tinystm::tmtype capacity;\n tinystm::tmtype sizeHM = 0;\n static constexpr double loadFactor = 0.75;\n tinystm::tmtype*> buckets; // An array of pointers to Nodes\n\n\npublic:\n TinySTMResizableHashSet(int maxThreads=0, uint64_t capacity=4) : capacity{capacity} {\n tinystm::updateTx([&] () {\n buckets = (tinystm::tmtype*)tinystm::tmMalloc(capacity*sizeof(tinystm::tmtype));\n for (int i = 0; i < capacity; i++) buckets[i] = nullptr;\n return true;\n });\n }\n\n\n ~TinySTMResizableHashSet() {\n tinystm::updateTx([&] () {\n for(int i = 0; i < capacity; i++){\n Node* node = buckets[i];\n while (node != nullptr) {\n Node* next = node->next;\n tinystm::tmDelete(node);\n node = next;\n }\n }\n tinystm::tmFree(buckets.load());\n return true;\n });\n }\n\n\n static std::string className() { return tinystm::TinySTM::className() + \"-HashMap\"; }\n\n\n void rebuild() {\n uint64_t newcapacity = 2*capacity;\n tinystm::tmtype* newbuckets = (tinystm::tmtype*)tinystm::tmMalloc(newcapacity*sizeof(tinystm::tmtype));\n for (int i = 0; i < newcapacity; i++) newbuckets[i] = nullptr;\n for (int i = 0; i < capacity; i++) {\n Node* node = buckets[i];\n while (node!=nullptr) {\n Node* next = node->next;\n auto h = std::hash{}(node->key) % newcapacity;\n node->next = newbuckets[h];\n newbuckets[h] = node;\n node = next;\n }\n }\n tinystm::tmFree(buckets);\n buckets = newbuckets;\n capacity = newcapacity;\n }\n\n\n /*\n * Adds a node with a key if the key is not present, otherwise replaces the value.\n * If saveOldValue is set, it will set 'oldValue' to the previous value, iff there was already a mapping.\n *\n * Returns true if there was no mapping for the key, false if there was already a value and it was replaced.\n */\n bool innerPut(const K& key) {\n if (sizeHM > capacity*loadFactor) rebuild();\n auto h = std::hash{}(key) % capacity;\n Node* node = buckets[h];\n Node* prev = node;\n while (true) {\n if (node == nullptr) {\n Node* newnode = tinystm::tmNew(key);\n if (node == prev) {\n buckets[h] = newnode;\n } else {\n prev->next = newnode;\n }\n sizeHM++;\n return true; // New insertion\n }\n if (key == node->key) return false;\n prev = node;\n node = node->next;\n }\n }\n\n\n /*\n * Removes a key and its mapping.\n * Saves the value in 'oldvalue' if 'saveOldValue' is set.\n *\n * Returns returns true if a matching key was found\n */\n bool innerRemove(const K& key) {\n auto h = std::hash{}(key) % capacity;\n Node* node = buckets[h];\n Node* prev = node;\n while (true) {\n if (node == nullptr) return false;\n if (key == node->key) {\n if (node == prev) {\n buckets[h] = node->next;\n } else {\n prev->next = node->next;\n }\n sizeHM--;\n tinystm::tmDelete(node);\n return true;\n }\n prev = node;\n node = node->next;\n }\n }\n\n\n /*\n * Returns true if key is present. Saves a copy of 'value' in 'oldValue' if 'saveOldValue' is set.\n */\n bool innerGet(const K& key) {\n auto h = std::hash{}(key) % capacity;\n Node* node = buckets[h];\n while (true) {\n if (node == nullptr) return false;\n if (key == node->key) return true;\n node = node->next;\n }\n }\n\n\n //\n // Set methods for running the usual tests and benchmarks\n //\n\n // Inserts a key only if it's not already present\n bool add(K key, const int tid=0) {\n return tinystm::updateTx([&] () {\n return innerPut(key);\n });\n }\n\n // Returns true only if the key was present\n bool remove(K key, const int tid=0) {\n return tinystm::updateTx([&] () {\n return innerRemove(key);\n });\n }\n\n bool contains(K key, const int tid=0) {\n return tinystm::readTx([&] () {\n return innerGet(key);\n });\n }\n\n // Used only for benchmarks\n void addAll(K** keys, const int size, const int tid=0) {\n for (int i = 0; i < size; i++) add(*keys[i]);\n }\n};\n\n#endif /* _TINY_STM_RESIZABLE_HASH_MAP_H_ */\n" }, { "path": "datastructures/linkedlists/CRWWPLinkedListSet.hpp", "content": "/******************************************************************************\r\n * Copyright (c) 2014-2018, Pedro Ramalhete, Andreia Correia\r\n * All rights reserved.\r\n *\r\n * Redistribution and use in source and binary forms, with or without\r\n * modification, are permitted provided that the following conditions are met:\r\n * * Redistributions of source code must retain the above copyright\r\n * notice, this list of conditions and the following disclaimer.\r\n * * Redistributions in binary form must reproduce the above copyright\r\n * notice, this list of conditions and the following disclaimer in the\r\n * documentation and/or other materials provided with the distribution.\r\n * * Neither the name of Concurrency Freaks nor the\r\n * names of its contributors may be used to endorse or promote products\r\n * derived from this software without specific prior written permission.\r\n *\r\n * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\r\n * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED\r\n * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\r\n * DISCLAIMED. IN NO EVENT SHALL BE LIABLE FOR ANY\r\n * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\r\n * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\r\n * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND\r\n * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\r\n * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\r\n * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\r\n ******************************************************************************\r\n */\r\n\r\n#ifndef _CRWWP_STM_LINKED_LIST_SET_H_\r\n#define _CRWWP_STM_LINKED_LIST_SET_H_\r\n\r\n#include \r\n#include \r\n#include \"stms/CRWWPSTM.hpp\"\r\n\r\n\r\n/**\r\n *

A Linked List Set for CRWWP STM (blocking)

\r\n *\r\n * TODO\r\n *\r\n *\r\n * @author Pedro Ramalhete\r\n * @author Andreia Correia\r\n */\r\ntemplate\r\nclass CRWWPLinkedListSet : public crwwpstm::tmbase {\r\n\r\nprivate:\r\n struct Node : public crwwpstm::tmbase {\r\n T key;\r\n crwwpstm::tmtype next {nullptr};\r\n Node() {}\r\n Node(T key) : key{key} { }\r\n };\r\n\r\n alignas(128) crwwpstm::tmtype head {nullptr};\r\n alignas(128) crwwpstm::tmtype tail {nullptr};\r\n\r\n\r\npublic:\r\n CRWWPLinkedListSet(unsigned int maxThreads=0) {\r\n Node* lhead = new Node();\r\n Node* ltail = new Node();\r\n head = lhead;\r\n head->next = ltail;\r\n tail = ltail;\r\n }\r\n\r\n\r\n ~CRWWPLinkedListSet() {\r\n // Delete all the nodes in the list\r\n Node* prev = head;\r\n Node* node = prev->next;\r\n while (node != tail) {\r\n delete prev;\r\n prev = node;\r\n node = node->next;\r\n }\r\n delete prev;\r\n delete tail;\r\n }\r\n\r\n\r\n static std::string className() { return crwwpstm::CRWWPSTM::className() + \"-LinkedListSet\"; }\r\n\r\n\r\n /*\r\n * Progress Condition: blocking\r\n * Adds a node with a key, returns false if the key is already in the set\r\n */\r\n bool add(T key, const int tid=0) {\r\n return crwwpstm::updateTx([&] () -> bool {\r\n Node* newNode = crwwpstm::tmNew(key);\r\n Node* prev = head;\r\n Node* node = prev->next;\r\n while (true) {\r\n if (node == tail) break;\r\n if (key == node->key) {\r\n crwwpstm::tmDelete(newNode); // If the key was already in the set, free the node that was never used\r\n return false;\r\n }\r\n if (node->key < key) break;\r\n prev = node;\r\n node = node->next;\r\n }\r\n prev->next = newNode;\r\n newNode->next = node;\r\n return true;\r\n });\r\n }\r\n\r\n\r\n /*\r\n * Progress Condition: blocking\r\n * Removes a node with an key, returns false if the key is not in the set\r\n */\r\n bool remove(T key, const int tid=0) {\r\n return crwwpstm::updateTx([&] () -> bool {\r\n Node* prev = head;\r\n Node* node = prev->next;\r\n while (true) {\r\n if (node == tail) return false;\r\n if (key == node->key) {\r\n prev->next = node->next;\r\n crwwpstm::tmDelete(node);\r\n return true;\r\n }\r\n if (node->key < key) return false;\r\n prev = node;\r\n node = node->next;\r\n }\r\n });\r\n }\r\n\r\n\r\n /*\r\n * Progress Condition: blocking\r\n * Returns true if it finds a node with a matching key\r\n */\r\n bool contains(T key, const int tid=0) {\r\n return crwwpstm::readTx([&] () -> bool {\r\n Node* node = head->next;\r\n while (true) {\r\n if (node == tail) return false;\r\n if (key == node->key) return true;\r\n if (node->key < key) return false;\r\n node = node->next;\r\n }\r\n });\r\n }\r\n\r\n\r\n bool addAll(T** keys, int size, const int tid) {\r\n for (int i = 0; i < size; i++) add(*keys[i], tid);\r\n }\r\n};\r\n\r\n#endif /* _C_RW_WP_STM_LINKED_LIST_SET_H_ */\r\n" }, { "path": "datastructures/linkedlists/ESTMLinkedListSet.hpp", "content": "/*\r\n * Copyright 2017-2018\r\n * Andreia Correia \r\n * Pedro Ramalhete \r\n * Pascal Felber \r\n * Nachshon Cohen \r\n *\r\n * This work is published under the MIT license. See LICENSE.txt\r\n */\r\n#ifndef _ESTM_LINKED_LIST_SET_H_\r\n#define _ESTM_LINKED_LIST_SET_H_\r\n\r\n\r\n#include \"../../stms/ESTM.hpp\" // This header defines the macros for the STM being compiled\r\n\r\n\r\n/**\r\n *

A Linked List Set for Elastic STM

\r\n * When we make the 'ltail' optimization here, it causes a crash on ESTM, therefore we don't do it.\r\n */\r\ntemplate\r\nclass ESTMLinkedListSet : public estm::tmbase {\r\n\r\nprivate:\r\n struct Node : public estm::tmbase {\r\n T key {};\r\n estm::tmtype next {nullptr};\r\n Node() {}\r\n Node(T key) : key{key} { }\r\n };\r\n\r\n alignas(128) estm::tmtype head {nullptr};\r\n alignas(128) estm::tmtype tail {nullptr};\r\n\r\n\r\npublic:\r\n ESTMLinkedListSet(unsigned int maxThreads=0) {\r\n estm::updateTx([&] () {\r\n Node* lhead = estm::tmNew();\r\n Node* ltail = estm::tmNew();\r\n head = lhead;\r\n head->next = ltail;\r\n tail = ltail;\r\n });\r\n }\r\n\r\n\r\n ~ESTMLinkedListSet() {\r\n estm::updateTx([&] () {\r\n // Delete all the nodes in the list\r\n Node* prev = head;\r\n Node* node = prev->next;\r\n while (node != tail) {\r\n estm::tmDelete(prev);\r\n prev = node;\r\n node = node->next;\r\n }\r\n estm::tmDelete(prev);\r\n estm::tmDelete(tail.load());\r\n });\r\n }\r\n\r\n\r\n static std::string className() { return estm::ESTM::className() + \"-LinkedListSet\"; }\r\n\r\n\r\n /*\r\n * Progress Condition: blocking\r\n * Adds a node with a key, returns false if the key is already in the set\r\n */\r\n bool add(T key, const int tid=0) {\r\n return estm::updateTx([this,key] () {\r\n Node* newNode = estm::tmNew(key);\r\n Node* prev = head;\r\n Node* node = prev->next;\r\n while (true) {\r\n if (node == tail) break;\r\n if (key == node->key) {\r\n estm::tmDelete(newNode); // If the key was already in the set, free the node that was never used\r\n return false;\r\n }\r\n if (node->key < key) break;\r\n prev = node;\r\n node = node->next;\r\n }\r\n prev->next = newNode;\r\n newNode->next = node;\r\n return true;\r\n });\r\n }\r\n\r\n\r\n /*\r\n * Progress Condition: blocking\r\n * Removes a node with an key, returns false if the key is not in the set\r\n */\r\n bool remove(T key, const int tid=0) {\r\n return estm::updateTx([this,key] () {\r\n Node* prev = head;\r\n Node* node = prev->next;\r\n while (true) {\r\n if (node == tail) return false;\r\n if (key == node->key) {\r\n prev->next = node->next;\r\n estm::tmDelete(node);\r\n return true;\r\n }\r\n if (node->key < key) return false;\r\n prev = node;\r\n node = node->next;\r\n }\r\n });\r\n }\r\n\r\n\r\n /*\r\n * Progress Condition: blocking\r\n * Returns true if it finds a node with a matching key\r\n */\r\n bool contains(T key, const int tid=0) {\r\n return estm::readTx([this,key] () {\r\n Node* node = head->next;\r\n while (true) {\r\n if (node == tail) return false;\r\n if (key == node->key) return true;\r\n if (node->key < key) return false;\r\n node = node->next;\r\n }\r\n });\r\n }\r\n\r\n\r\n bool addAll(T** keys, int size, const int tid) {\r\n for (int i = 0; i < size; i++) add(*keys[i], tid);\r\n return true;\r\n }\r\n};\r\n\r\n#endif /* _ESTM_LINKED_LIST_SET_H_ */\r\n" }, { "path": "datastructures/linkedlists/HazardEras.hpp", "content": "/******************************************************************************\r\n * Copyright (c) 2016-2017, Pedro Ramalhete, Andreia Correia\r\n * All rights reserved.\r\n *\r\n * Redistribution and use in source and binary forms, with or without\r\n * modification, are permitted provided that the following conditions are met:\r\n * * Redistributions of source code must retain the above copyright\r\n * notice, this list of conditions and the following disclaimer.\r\n * * Redistributions in binary form must reproduce the above copyright\r\n * notice, this list of conditions and the following disclaimer in the\r\n * documentation and/or other materials provided with the distribution.\r\n * * Neither the name of Concurrency Freaks nor the\r\n * names of its contributors may be used to endorse or promote products\r\n * derived from this software without specific prior written permission.\r\n\r\n *\r\n * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\r\n * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED\r\n * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\r\n * DISCLAIMED. IN NO EVENT SHALL BE LIABLE FOR ANY\r\n * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\r\n * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\r\n * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND\r\n * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\r\n * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\r\n * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\r\n ******************************************************************************\r\n */\r\n\r\n#ifndef _HAZARD_ERAS_H_\r\n#define _HAZARD_ERAS_H_\r\n\r\n#include \r\n#include \r\n#include \r\n#include \r\n\r\n/*\r\n *

Hazard Eras

\r\n * This a light-weight implementation of hazard eras, where each thread has a\r\n * thread-local list of retired objects.\r\n *\r\n * This is based on the paper \"Hazard Eras - Non-Blocking Memory Reclamation\"\r\n * by Pedro Ramalhete and Andreia Correia:\r\n * https://github.com/pramalhe/ConcurrencyFreaks/blob/master/papers/hazarderas-2017.pdf\r\n *\r\n * The type T is for the objects/nodes and it's it must have the members newEra, delEra\r\n *\r\n * R is zero.\r\n *\r\n *

\r\n * @author Pedro Ramalhete\r\n * @author Andreia Correia\r\n */\r\ntemplate\r\nclass HazardEras {\r\n\r\nprivate:\r\n static const uint64_t NONE = 0;\r\n static const int HE_MAX_THREADS = 128;\r\n static const int MAX_HES = 5; // This is named 'K' in the HP paper\r\n static const int CLPAD = 128/sizeof(std::atomic);\r\n static const int HE_THRESHOLD_R = 0; // This is named 'R' in the HP paper\r\n\r\n const int maxHEs;\r\n const int maxThreads;\r\n\r\n alignas(128) std::atomic eraClock {1};\r\n alignas(128) std::atomic* he[HE_MAX_THREADS];\r\n // It's not nice that we have a lot of empty vectors, but we need padding to avoid false sharing\r\n alignas(128) std::vector retiredList[HE_MAX_THREADS*CLPAD];\r\n\r\npublic:\r\n HazardEras(int maxHEs=MAX_HES, int maxThreads=HE_MAX_THREADS) : maxHEs{maxHEs}, maxThreads{maxThreads} {\r\n for (int it = 0; it < HE_MAX_THREADS; it++) {\r\n he[it] = new std::atomic[CLPAD*2]; // We allocate four cache lines to allow for many hps and without false sharing\r\n retiredList[it*CLPAD].reserve(maxThreads*maxHEs);\r\n for (int ihe = 0; ihe < MAX_HES; ihe++) {\r\n he[it][ihe].store(NONE, std::memory_order_relaxed);\r\n }\r\n }\r\n static_assert(std::is_same::value, \"T::newEra must be uint64_t\");\r\n static_assert(std::is_same::value, \"T::delEra must be uint64_t\");\r\n }\r\n\r\n ~HazardEras() {\r\n for (int it = 0; it < HE_MAX_THREADS; it++) {\r\n delete[] he[it];\r\n // Clear the current retired nodes\r\n for (unsigned iret = 0; iret < retiredList[it*CLPAD].size(); iret++) {\r\n delete retiredList[it*CLPAD][iret];\r\n }\r\n }\r\n }\r\n\r\n\r\n inline uint64_t getEra() {\r\n return eraClock.load();\r\n }\r\n\r\n\r\n /**\r\n * Progress Condition: wait-free bounded (by maxHEs)\r\n */\r\n inline void clear(const int tid) {\r\n for (int ihe = 0; ihe < maxHEs; ihe++) {\r\n he[tid][ihe].store(NONE, std::memory_order_release);\r\n }\r\n }\r\n\r\n\r\n /**\r\n * Progress Condition: lock-free\r\n */\r\n inline T* get_protected(int index, const std::atomic& atom, const int tid) {\r\n auto prevEra = he[tid][index].load(std::memory_order_relaxed);\r\n\t\twhile (true) {\r\n\t\t T* ptr = atom.load();\r\n\t\t auto era = eraClock.load(std::memory_order_acquire);\r\n\t\t if (era == prevEra) return ptr;\r\n he[tid][index].store(era);\r\n prevEra = era;\r\n\t\t}\r\n }\r\n\r\n inline void protectEraRelease(int index, int other, const int tid) {\r\n auto era = he[tid][other].load(std::memory_order_relaxed);\r\n if (he[tid][index].load(std::memory_order_relaxed) == era) return;\r\n he[tid][index].store(era, std::memory_order_release);\r\n }\r\n\r\n\r\n /*\r\n * Does a single iteration. Must be integrated into the algorithm that's using HE.\r\n * In other words, we must re-check if era has changed\r\n *\r\n * Progress Condition: wait-free population oblivious\r\n */\r\n inline T* protectPtr(int index, const std::atomic& atom, uint64_t& prevEra, const int tid) {\r\n T* ptr = atom.load(std::memory_order_acquire);\r\n auto era = eraClock.load();\r\n if (prevEra != era) {\r\n prevEra = era;\r\n he[tid][index].store(era, std::memory_order_relaxed);\r\n std::atomic_thread_fence(std::memory_order_seq_cst);\r\n }\r\n return ptr;\r\n }\r\n\r\n\r\n /**\r\n * Retire an object (node)\r\n * Progress Condition: wait-free bounded\r\n *\r\n * Doing rlist.erase() is not the most efficient way to remove entries from a std::vector, but ok...\r\n */\r\n void retire(T* ptr, const int mytid) {\r\n auto currEra = eraClock.load();\r\n ptr->delEra = currEra;\r\n auto& rlist = retiredList[mytid*CLPAD];\r\n rlist.push_back(ptr);\r\n if (eraClock == currEra) eraClock.fetch_add(1);\r\n for (unsigned iret = 0; iret < rlist.size();) {\r\n auto obj = rlist[iret];\r\n if (canDelete(obj, mytid)) {\r\n rlist.erase(rlist.begin() + iret);\r\n delete obj;\r\n continue;\r\n }\r\n iret++;\r\n }\r\n }\r\n\r\nprivate:\r\n bool canDelete(T* obj, const int mytid) {\r\n for (int tid = 0; tid < maxThreads; tid++) {\r\n for (int ihe = 0; ihe < maxHEs; ihe++) {\r\n const auto era = he[tid][ihe].load(std::memory_order_acquire);\r\n if (era == NONE || era < obj->newEra || era > obj->delEra) continue;\r\n return false;\r\n }\r\n }\r\n return true;\r\n }\r\n\r\n};\r\n\r\n#endif /* _HAZARD_ERAS_H_ */\r\n" }, { "path": "datastructures/linkedlists/HazardPointers.hpp", "content": "/******************************************************************************\r\n * Copyright (c) 2014-2017, Pedro Ramalhete, Andreia Correia\r\n * All rights reserved.\r\n *\r\n * Redistribution and use in source and binary forms, with or without\r\n * modification, are permitted provided that the following conditions are met:\r\n * * Redistributions of source code must retain the above copyright\r\n * notice, this list of conditions and the following disclaimer.\r\n * * Redistributions in binary form must reproduce the above copyright\r\n * notice, this list of conditions and the following disclaimer in the\r\n * documentation and/or other materials provided with the distribution.\r\n * * Neither the name of Concurrency Freaks nor the\r\n * names of its contributors may be used to endorse or promote products\r\n * derived from this software without specific prior written permission.\r\n\r\n *\r\n * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\r\n * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED\r\n * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\r\n * DISCLAIMED. IN NO EVENT SHALL BE LIABLE FOR ANY\r\n * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\r\n * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\r\n * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND\r\n * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\r\n * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\r\n * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\r\n ******************************************************************************\r\n */\r\n\r\n#ifndef _HAZARD_POINTERS_H_\r\n#define _HAZARD_POINTERS_H_\r\n\r\n#include \r\n#include \r\n#include \r\n\r\n\r\ntemplate\r\nclass HazardPointers {\r\n\r\nprivate:\r\n static const int HP_MAX_THREADS = 128;\r\n static const int HP_MAX_HPS = 5; // This is named 'K' in the HP paper\r\n static const int CLPAD = 128/sizeof(std::atomic);\r\n static const int HP_THRESHOLD_R = 0; // This is named 'R' in the HP paper\r\n static const int MAX_RETIRED = HP_MAX_THREADS*HP_MAX_HPS; // Maximum number of retired objects per thread\r\n\r\n const int maxHPs;\r\n const int maxThreads;\r\n\r\n alignas(128) std::atomic* hp[HP_MAX_THREADS];\r\n // It's not nice that we have a lot of empty vectors, but we need padding to avoid false sharing\r\n alignas(128) std::vector retiredList[HP_MAX_THREADS*CLPAD];\r\n\r\npublic:\r\n HazardPointers(int maxHPs=HP_MAX_HPS, int maxThreads=HP_MAX_THREADS) : maxHPs{maxHPs}, maxThreads{maxThreads} {\r\n for (int it = 0; it < HP_MAX_THREADS; it++) {\r\n hp[it] = new std::atomic[CLPAD*2]; // We allocate four cache lines to allow for many hps and without false sharing\r\n retiredList[it*CLPAD].reserve(MAX_RETIRED);\r\n for (int ihp = 0; ihp < HP_MAX_HPS; ihp++) {\r\n hp[it][ihp].store(nullptr, std::memory_order_relaxed);\r\n }\r\n }\r\n }\r\n\r\n ~HazardPointers() {\r\n for (int it = 0; it < HP_MAX_THREADS; it++) {\r\n delete[] hp[it];\r\n // Clear the current retired nodes\r\n for (unsigned iret = 0; iret < retiredList[it*CLPAD].size(); iret++) {\r\n delete retiredList[it*CLPAD][iret];\r\n }\r\n }\r\n }\r\n\r\n\r\n /**\r\n * Progress Condition: wait-free bounded (by maxHPs)\r\n */\r\n inline void clear(const int tid) {\r\n for (int ihp = 0; ihp < maxHPs; ihp++) {\r\n hp[tid][ihp].store(nullptr, std::memory_order_release);\r\n }\r\n }\r\n\r\n\r\n /**\r\n * Progress Condition: wait-free population oblivious\r\n */\r\n inline void clearOne(int ihp, const int tid) {\r\n hp[tid][ihp].store(nullptr, std::memory_order_release);\r\n }\r\n\r\n\r\n /**\r\n * Progress Condition: lock-free\r\n */\r\n inline T* protect(int index, const std::atomic& atom, const int tid) {\r\n T* n = nullptr;\r\n T* ret;\r\n\t\twhile ((ret = atom.load()) != n) {\r\n\t\t\thp[tid][index].store(ret);\r\n\t\t\tn = ret;\r\n\t\t}\r\n\t\treturn ret;\r\n }\r\n\r\n\t\r\n /**\r\n * This returns the same value that is passed as ptr, which is sometimes useful\r\n * Progress Condition: wait-free population oblivious\r\n */\r\n inline T* protectPtr(int index, T* ptr, const int tid) {\r\n hp[tid][index].store(ptr);\r\n /*\r\n // For x86-only implementations, use this instead (it's 2x faster than mfence on x86):\r\n hp[tid][index].store(ptr, std::memory_order_release);\r\n __asm__ __volatile__ (\"lock;addl $0,(%%rsp);\" ::: \"cc\",\"memory\") ;\r\n */\r\n return ptr;\r\n }\r\n\r\n\r\n\r\n /**\r\n * This returns the same value that is passed as ptr, which is sometimes useful\r\n * Progress Condition: wait-free population oblivious\r\n */\r\n inline T* protectPtrRelease(int index, T* ptr, const int tid) {\r\n hp[tid][index].store(ptr, std::memory_order_release);\r\n return ptr;\r\n }\r\n\r\n\r\n /**\r\n * Progress Condition: wait-free bounded (by the number of threads squared)\r\n */\r\n void retire(T* ptr, const int tid) {\r\n retiredList[tid*CLPAD].push_back(ptr);\r\n if (retiredList[tid*CLPAD].size() < HP_THRESHOLD_R) return;\r\n for (unsigned iret = 0; iret < retiredList[tid*CLPAD].size();) {\r\n auto obj = retiredList[tid*CLPAD][iret];\r\n bool canDelete = true;\r\n for (int tid = 0; tid < maxThreads && canDelete; tid++) {\r\n for (int ihp = 0; ihp < maxHPs; ihp++) {\r\n if (hp[tid][ihp].load() == obj) {\r\n canDelete = false;\r\n break;\r\n }\r\n }\r\n }\r\n if (canDelete) {\r\n retiredList[tid*CLPAD].erase(retiredList[tid*CLPAD].begin() + iret);\r\n delete obj;\r\n continue;\r\n }\r\n iret++;\r\n }\r\n }\r\n};\r\n\r\n#endif /* _HAZARD_POINTERS_H_ */\r\n" }, { "path": "datastructures/linkedlists/MagedHarrisLinkedListSetHE.hpp", "content": "/******************************************************************************\r\n * Copyright (c) 2014-2016, Pedro Ramalhete, Andreia Correia\r\n * All rights reserved.\r\n *\r\n * Redistribution and use in source and binary forms, with or without\r\n * modification, are permitted provided that the following conditions are met:\r\n * * Redistributions of source code must retain the above copyright\r\n * notice, this list of conditions and the following disclaimer.\r\n * * Redistributions in binary form must reproduce the above copyright\r\n * notice, this list of conditions and the following disclaimer in the\r\n * documentation and/or other materials provided with the distribution.\r\n * * Neither the name of Concurrency Freaks nor the\r\n * names of its contributors may be used to endorse or promote products\r\n * derived from this software without specific prior written permission.\r\n *\r\n * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\r\n * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED\r\n * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\r\n * DISCLAIMED. IN NO EVENT SHALL BE LIABLE FOR ANY\r\n * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\r\n * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\r\n * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND\r\n * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\r\n * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\r\n * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\r\n ******************************************************************************\r\n */\r\n\r\n#ifndef _MAGED_M_MICHAEL_LINKED_LIST_HE_H_\r\n#define _MAGED_M_MICHAEL_LINKED_LIST_HE_H_\r\n\r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \"common/HazardEras.hpp\"\r\n\r\n\r\n\r\n/**\r\n * This is the linked list by Maged M. Michael that uses Hazard Eras.\r\n * Lock-Free Linked List as described in Maged M. Michael paper (Figure 7):\r\n * http://www.cs.tau.ac.il/~afek/p73-Lock-Free-HashTbls-michael.pdf\r\n *\r\n *

\r\n * This set has three operations:\r\n *

    \r\n *
  • add(x) - Lock-Free\r\n *
  • remove(x) - Lock-Free\r\n *
  • contains(x) - Lock-Free\r\n *

\r\n *

\r\n * @author Pedro Ramalhete\r\n * @author Andreia Correia\r\n */\r\ntemplate\r\nclass MagedHarrisLinkedListSetHE {\r\n\r\nprivate:\r\n struct Node {\r\n T key;\r\n uint64_t newEra;\r\n uint64_t delEra;\r\n std::atomic next;\r\n\r\n Node(T key, uint64_t newEra) : key{key}, newEra{newEra}, delEra{0}, next{nullptr} { }\r\n\r\n bool casNext(Node *cmp, Node *val) {\r\n return next.compare_exchange_strong(cmp, val);\r\n }\r\n };\r\n\r\n // Pointers to head and tail sentinel nodes of the list\r\n std::atomic head;\r\n std::atomic tail;\r\n\r\n const int maxThreads;\r\n\r\n HazardEras he {3, maxThreads};\r\n const int kHp0 = 0; // Protects next\r\n const int kHp1 = 1; // Protects curr\r\n const int kHp2 = 2; // Protects prev\r\n\r\npublic:\r\n\r\n MagedHarrisLinkedListSetHE(const int maxThreads) : maxThreads{maxThreads} {\r\n head.store(new Node({}, 1)); // Uses K's default constructor\r\n tail.store(new Node({}, 1)); // Uses K's default constructor\r\n head.load()->next.store(tail.load());\r\n }\r\n\r\n\r\n // We don't expect the destructor to be called if this instance can still be in use\r\n ~MagedHarrisLinkedListSetHE() {\r\n Node *prev = head.load();\r\n Node *node = prev->next.load();\r\n while (node != nullptr) {\r\n delete prev;\r\n prev = node;\r\n node = prev->next.load();\r\n }\r\n delete prev;\r\n }\r\n\r\n static std::string className() { return \"MagedHarris-LinkedListSetHE\"; }\r\n\r\n /*\r\n * This function is single threaded to be called at the start of the test.\r\n * It is assumed keys are ordered\r\n */\r\n void addAll(T** keys, const int size, const int tid) {\r\n Node* node = head;\r\n for(int i=0;inext.store(newNode, std::memory_order_relaxed);\r\n node = newNode;\r\n }\r\n node->next.store(tail.load(std::memory_order_relaxed), std::memory_order_relaxed);\r\n }\r\n\r\n /**\r\n * This method is named 'Insert()' in the original paper.\r\n * Taken from Figure 7 of the paper:\r\n * \"High Performance Dynamic Lock-Free Hash Tables and List-Based Sets\"\r\n *

\r\n * Progress Condition: Lock-Free\r\n *\r\n */\r\n bool add(T key, const int tid)\r\n {\r\n Node *curr, *next;\r\n std::atomic *prev;\r\n Node* newNode = new Node(key, he.getEra());\r\n while (true) {\r\n if (find(&key, &prev, &curr, &next, tid)) {\r\n delete newNode; // There is already a matching key\r\n he.clear(tid);\r\n return false;\r\n }\r\n newNode->next.store(curr, std::memory_order_relaxed);\r\n Node *tmp = getUnmarked(curr);\r\n if (prev->compare_exchange_strong(tmp, newNode)) { // seq-cst\r\n he.clear(tid);\r\n return true;\r\n }\r\n }\r\n }\r\n\r\n\r\n /**\r\n * This method is named 'Delete()' in the original paper.\r\n * Taken from Figure 7 of the paper:\r\n * \"High Performance Dynamic Lock-Free Hash Tables and List-Based Sets\"\r\n */\r\n bool remove(T key, const int tid)\r\n {\r\n Node *curr, *next;\r\n std::atomic *prev;\r\n while (true) {\r\n /* Try to find the key in the list. */\r\n if (!find(&key, &prev, &curr, &next, tid)) {\r\n he.clear(tid);\r\n return false;\r\n }\r\n /* Mark if needed. */\r\n Node *tmp = getUnmarked(next);\r\n if (!curr->next.compare_exchange_strong(tmp, getMarked(next))) {\r\n continue; /* Another thread interfered. */\r\n }\r\n\r\n tmp = getUnmarked(curr);\r\n if (prev->compare_exchange_strong(tmp, getUnmarked(next))) { /* Unlink */\r\n he.clear(tid);\r\n he.retire(getUnmarked(curr), tid); /* Reclaim */\r\n } else {\r\n he.clear(tid);\r\n }\r\n /*\r\n * If we want to prevent the possibility of there being an\r\n * unbounded number of unmarked nodes, add \"else _find(head,key).\"\r\n * This is not necessary for correctness.\r\n */\r\n return true;\r\n }\r\n }\r\n\r\n\r\n /**\r\n * This is named 'Search()' on the original paper\r\n * Taken from Figure 7 of the paper:\r\n * \"High Performance Dynamic Lock-Free Hash Tables and List-Based Sets\"\r\n *

\r\n * Progress Condition: Lock-Free\r\n */\r\n bool contains(T key, const int tid)\r\n {\r\n Node *curr, *next;\r\n std::atomic *prev;\r\n bool isContains = find(&key, &prev, &curr, &next, tid);\r\n he.clear(tid);\r\n return isContains;\r\n }\r\n\r\n\r\nprivate:\r\n\r\n /**\r\n *

\r\n * Progress Condition: Lock-Free\r\n */\r\n bool find (T* key, std::atomic **par_prev, Node **par_curr, Node **par_next, const int tid)\r\n {\r\n std::atomic *prev;\r\n Node *curr, *next;\r\n\r\n try_again:\r\n prev = &head;\r\n // Protect curr with a hazard era\r\n curr = he.get_protected(kHp1, *prev, tid);\r\n while (true) {\r\n if (getUnmarked(curr) == nullptr) break; // TODO: Will it ever happen?\r\n // Protect next with a hazard era.\r\n next = he.get_protected(kHp0, curr->next, tid);\r\n if (getUnmarked(curr)->next.load() != next) goto try_again;\r\n if (getUnmarked(next) == tail.load()) break;\r\n if (prev->load() != getUnmarked(curr)) goto try_again;\r\n if (getUnmarked(next) == next) { // !cmark in the paper\r\n if (!(getUnmarked(curr)->key < *key)) { // Check for null to handle head and tail\r\n *par_curr = curr;\r\n *par_prev = prev;\r\n *par_next = next;\r\n return (getUnmarked(curr)->key == *key);\r\n }\r\n prev = &getUnmarked(curr)->next;\r\n he.protectEraRelease(kHp2, kHp1, tid);\r\n } else {\r\n // Update the link and retire the node.\r\n Node *tmp = getUnmarked(curr);\r\n if (!prev->compare_exchange_strong(tmp, getUnmarked(next))) {\r\n goto try_again;\r\n }\r\n he.retire(getUnmarked(curr), tid);\r\n }\r\n curr = next;\r\n he.protectEraRelease(kHp1, kHp0, tid);\r\n }\r\n *par_curr = curr;\r\n *par_prev = prev;\r\n *par_next = next;\r\n return false;\r\n }\r\n\r\n bool isMarked(Node * node) {\r\n \treturn ((size_t) node & 0x1);\r\n }\r\n\r\n Node * getMarked(Node * node) {\r\n \treturn (Node*)((size_t) node | 0x1);\r\n }\r\n\r\n Node * getUnmarked(Node * node) {\r\n \treturn (Node*)((size_t) node & (~0x1));\r\n }\r\n};\r\n\r\n#endif /* _MAGED_M_MICHAEL_LINKED_LIST_HE_H_ */\r\n" }, { "path": "datastructures/linkedlists/MagedHarrisLinkedListSetHP.hpp", "content": "/******************************************************************************\r\n * Copyright (c) 2014-2016, Pedro Ramalhete, Andreia Correia\r\n * All rights reserved.\r\n *\r\n * Redistribution and use in source and binary forms, with or without\r\n * modification, are permitted provided that the following conditions are met:\r\n * * Redistributions of source code must retain the above copyright\r\n * notice, this list of conditions and the following disclaimer.\r\n * * Redistributions in binary form must reproduce the above copyright\r\n * notice, this list of conditions and the following disclaimer in the\r\n * documentation and/or other materials provided with the distribution.\r\n * * Neither the name of Concurrency Freaks nor the\r\n * names of its contributors may be used to endorse or promote products\r\n * derived from this software without specific prior written permission.\r\n *\r\n * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\r\n * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED\r\n * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\r\n * DISCLAIMED. IN NO EVENT SHALL BE LIABLE FOR ANY\r\n * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\r\n * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\r\n * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND\r\n * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\r\n * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\r\n * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\r\n ******************************************************************************\r\n */\r\n\r\n#ifndef _MAGED_MICHAEL_TIM_HARRIS_LINKED_LIST_HP_H_\r\n#define _MAGED_MICHAEL_TIM_HARRIS_LINKED_LIST_HP_H_\r\n\r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \"common/HazardPointers.hpp\"\r\n\r\n\r\n\r\n/**\r\n * This is the linked list by Maged M. Michael that uses Hazard Pointers in\r\n * a correct way because Harris original algorithm with HPs doesn't.\r\n * Lock-Free Linked List as described in Maged M. Michael paper (Figure 4):\r\n * http://www.cs.tau.ac.il/~afek/p73-Lock-Free-HashTbls-michael.pdf\r\n *\r\n * \r\n *

\r\n * This set has three operations:\r\n *

    \r\n *
  • add(x) - Lock-Free\r\n *
  • remove(x) - Lock-Free\r\n *
  • contains(x) - Lock-Free\r\n *

\r\n *

\r\n * @author Pedro Ramalhete\r\n * @author Andreia Correia\r\n */\r\ntemplate \r\nclass MagedHarrisLinkedListSetHP {\r\n\r\nprivate:\r\n struct Node {\r\n T key;\r\n std::atomic next;\r\n\r\n Node(T key) : key{key}, next{nullptr} { }\r\n\r\n bool casNext(Node *cmp, Node *val) {\r\n return next.compare_exchange_strong(cmp, val);\r\n }\r\n };\r\n\r\n // Pointers to head and tail sentinel nodes of the list\r\n std::atomic head;\r\n std::atomic tail;\r\n\r\n const int maxThreads;\r\n\r\n // We need 3 hazard pointers\r\n HazardPointers hp {3, maxThreads};\r\n const int kHp0 = 0; // Protects next\r\n const int kHp1 = 1; // Protects curr\r\n const int kHp2 = 2; // Protects prev\r\n\r\npublic:\r\n\r\n MagedHarrisLinkedListSetHP(const int maxThreads) : maxThreads{maxThreads} {\r\n head.store(new Node({}));\r\n tail.store(new Node({}));\r\n head.load()->next.store(tail.load());\r\n }\r\n\r\n\r\n // We don't expect the destructor to be called if this instance can still be in use\r\n ~MagedHarrisLinkedListSetHP() {\r\n Node *prev = head.load();\r\n Node *node = prev->next.load();\r\n while (node != nullptr) {\r\n delete prev;\r\n prev = node;\r\n node = prev->next.load();\r\n }\r\n delete prev;\r\n }\r\n\r\n static std::string className() { return \"MagedHarris-LinkedListSetHP\"; }\r\n\r\n /*\r\n * This function is single threaded to be called at the start of the test.\r\n * It is assumed keys are ordered\r\n */\r\n void addAll(T** keys, const int size, const int tid) {\r\n Node* node = head;\r\n for(int i=0;inext.store(newNode, std::memory_order_relaxed);\r\n node = newNode;\r\n }\r\n node->next.store(tail.load(std::memory_order_relaxed), std::memory_order_relaxed);\r\n }\r\n\r\n /**\r\n * This method is named 'Insert()' in the original paper.\r\n * Taken from Figure 7 of the paper:\r\n * \"High Performance Dynamic Lock-Free Hash Tables and List-Based Sets\"\r\n *

\r\n * Progress Condition: Lock-Free\r\n *\r\n */\r\n bool add(T key, const int tid)\r\n {\r\n Node *curr, *next;\r\n std::atomic *prev;\r\n Node* newNode = new Node(key);\r\n while (true) {\r\n if (find(&key, &prev, &curr, &next, tid)) {\r\n delete newNode; // There is already a matching key\r\n hp.clear(tid);\r\n return false;\r\n }\r\n newNode->next.store(curr, std::memory_order_relaxed);\r\n Node *tmp = getUnmarked(curr);\r\n if (prev->compare_exchange_strong(tmp, newNode)) { // seq-cst\r\n hp.clear(tid);\r\n return true;\r\n }\r\n }\r\n }\r\n\r\n\r\n /**\r\n * This method is named 'Delete()' in the original paper.\r\n * Taken from Figure 7 of the paper:\r\n * \"High Performance Dynamic Lock-Free Hash Tables and List-Based Sets\"\r\n */\r\n bool remove(T key, const int tid)\r\n {\r\n Node *curr, *next;\r\n std::atomic *prev;\r\n while (true) {\r\n /* Try to find the key in the list. */\r\n if (!find(&key, &prev, &curr, &next, tid)) {\r\n hp.clear(tid);\r\n return false;\r\n }\r\n /* Mark if needed. */\r\n Node *tmp = getUnmarked(next);\r\n if (!curr->next.compare_exchange_strong(tmp, getMarked(next))) {\r\n continue; /* Another thread interfered. */\r\n }\r\n\r\n tmp = getUnmarked(curr);\r\n if (prev->compare_exchange_strong(tmp, getUnmarked(next))) { /* Unlink */\r\n hp.clear(tid);\r\n hp.retire(getUnmarked(curr), tid); /* Reclaim */\r\n } else {\r\n hp.clear(tid);\r\n }\r\n /*\r\n * If we want to prevent the possibility of there being an\r\n * unbounded number of unmarked nodes, add \"else _find(head,key).\"\r\n * This is not necessary for correctness.\r\n */\r\n return true;\r\n }\r\n }\r\n\r\n\r\n /**\r\n * This is named 'Search()' on the original paper\r\n * Taken from Figure 7 of the paper:\r\n * \"High Performance Dynamic Lock-Free Hash Tables and List-Based Sets\"\r\n *

\r\n * Progress Condition: Lock-Free\r\n */\r\n bool contains(T key, const int tid)\r\n {\r\n Node *curr, *next;\r\n std::atomic *prev;\r\n bool isContains = find(&key, &prev, &curr, &next, tid);\r\n hp.clear(tid);\r\n return isContains;\r\n }\r\n\r\n\r\nprivate:\r\n\r\n /**\r\n * TODO: This needs to be code reviewed... it's not production-ready\r\n *

\r\n * Progress Condition: Lock-Free\r\n */\r\n bool find (T* key, std::atomic **par_prev, Node **par_curr, Node **par_next, const int tid)\r\n {\r\n std::atomic *prev;\r\n Node *curr, *next;\r\n\r\n try_again:\r\n prev = &head;\r\n curr = prev->load();\r\n // Protect curr with a hazard pointer.\r\n hp.protectPtr(kHp1, curr, tid);\r\n if (prev->load() != getUnmarked(curr)) goto try_again;\r\n while (true) {\r\n if (getUnmarked(curr) == nullptr) break;\r\n // Protect next with a hazard pointer.\r\n next = curr->next.load();\r\n hp.protectPtr(kHp0, getUnmarked(next), tid);\r\n if (getUnmarked(curr)->next.load() != next) goto try_again;\r\n if (getUnmarked(next) == tail.load()) break;\r\n if (prev->load() != getUnmarked(curr)) goto try_again;\r\n if (getUnmarked(next) == next) { // !cmark in the paper\r\n if (!(getUnmarked(curr)->key < *key)) { // Check for null to handle head and tail\r\n *par_curr = curr;\r\n *par_prev = prev;\r\n *par_next = next;\r\n return (getUnmarked(curr)->key == *key);\r\n }\r\n prev = &getUnmarked(curr)->next;\r\n hp.protectRelease(kHp2, getUnmarked(curr), tid);\r\n } else {\r\n // Update the link and retire the node.\r\n Node *tmp = getUnmarked(curr);\r\n if (!prev->compare_exchange_strong(tmp, getUnmarked(next))) {\r\n goto try_again;\r\n }\r\n hp.retire(getUnmarked(curr), tid);\r\n }\r\n curr = next;\r\n hp.protectRelease(kHp1, getUnmarked(next), tid);\r\n }\r\n *par_curr = curr;\r\n *par_prev = prev;\r\n *par_next = next;\r\n return false;\r\n }\r\n\r\n bool isMarked(Node * node) {\r\n \treturn ((size_t) node & 0x1);\r\n }\r\n\r\n Node * getMarked(Node * node) {\r\n \treturn (Node*)((size_t) node | 0x1);\r\n }\r\n\r\n Node * getUnmarked(Node * node) {\r\n \treturn (Node*)((size_t) node & (~0x1));\r\n }\r\n};\r\n\r\n#endif /* _MAGED_MICHAEL_TIM_HARRIS_LINKED_LIST_HP_H_ */\r\n" }, { "path": "datastructures/linkedlists/OFLFLinkedListSet.hpp", "content": "/*\r\n * Copyright 2017-2018\r\n * Andreia Correia \r\n * Pedro Ramalhete \r\n * Pascal Felber \r\n * Nachshon Cohen \r\n *\r\n * This work is published under the MIT license. See LICENSE.txt\r\n */\r\n#ifndef _ONEFILE_LF_LINKED_LIST_SET_H_\r\n#define _ONEFILE_LF_LINKED_LIST_SET_H_\r\n\r\n#include \r\n#include \r\n#include \"stms/OneFileLF.hpp\"\r\n\r\n\r\n/**\r\n *

A Linked List Set for One-File STM (Lock-Free)

\r\n */\r\ntemplate\r\nclass OFLFLinkedListSet : public oflf::tmbase {\r\n\r\nprivate:\r\n struct Node : public oflf::tmbase {\r\n T key {};\r\n oflf::tmtype next {nullptr};\r\n Node() {}\r\n Node(T key) : key{key} { }\r\n };\r\n\r\n alignas(128) oflf::tmtype head {nullptr};\r\n alignas(128) oflf::tmtype tail {nullptr};\r\n\r\n\r\npublic:\r\n OFLFLinkedListSet(unsigned int maxThreads=0) {\r\n oflf::updateTx([this] () {\r\n Node* lhead = oflf::tmNew();\r\n Node* ltail = oflf::tmNew();\r\n head = lhead;\r\n head->next = ltail;\r\n tail = ltail;\r\n });\r\n }\r\n\r\n\r\n ~OFLFLinkedListSet() {\r\n oflf::updateTx([this] () {\r\n // Delete all the nodes in the list\r\n Node* prev = head;\r\n Node* node = prev->next;\r\n while (node != tail) {\r\n oflf::tmDelete(prev);\r\n prev = node;\r\n node = node->next;\r\n }\r\n oflf::tmDelete(prev);\r\n oflf::tmDelete(tail.pload());\r\n });\r\n }\r\n\r\n\r\n static std::string className() { return oflf::OneFileLF::className() + \"-LinkedListSet\"; }\r\n\r\n\r\n /*\r\n * Progress Condition: lock-free\r\n * Adds a node with a key, returns false if the key is already in the set\r\n */\r\n bool add(T key, const int tid=0) {\r\n return oflf::updateTx([this,key] () -> bool {\r\n Node* newNode = oflf::tmNew(key);\r\n Node* prev = head;\r\n Node* node = prev->next;\r\n Node* ltail = tail;\r\n while (true) {\r\n if (node == ltail) break;\r\n T nkey = node->key;\r\n if (key == nkey) {\r\n oflf::tmDelete(newNode); // If the key was already in the set, free the node that was never used\r\n return false;\r\n }\r\n if (nkey < key) break;\r\n prev = node;\r\n node = node->next;\r\n }\r\n prev->next = newNode;\r\n newNode->next = node;\r\n return true;\r\n });\r\n }\r\n\r\n\r\n /*\r\n * Progress Condition: lock-free\r\n * Removes a node with an key, returns false if the key is not in the set\r\n */\r\n bool remove(T key, const int tid=0) {\r\n return oflf::updateTx([this,key] () -> bool {\r\n Node* prev = head;\r\n Node* node = prev->next;\r\n Node* ltail = tail;\r\n while (true) {\r\n if (node == ltail) return false;\r\n T nkey = node->key;\r\n if (key == nkey) {\r\n prev->next = node->next;\r\n oflf::tmDelete(node);\r\n return true;\r\n }\r\n if (nkey < key) return false;\r\n prev = node;\r\n node = node->next;\r\n }\r\n });\r\n }\r\n\r\n\r\n /*\r\n * Progress Condition: lock-free\r\n * Returns true if it finds a node with a matching key\r\n */\r\n bool contains(T key, const int tid=0) {\r\n return oflf::readTx([this,key] () -> bool {\r\n Node* node = head->next;\r\n Node* ltail = tail;\r\n while (true) {\r\n if (node == ltail) return false;\r\n T nkey = node->key;\r\n if (key == nkey) return true;\r\n if (nkey < key) return false;\r\n node = node->next;\r\n }\r\n });\r\n }\r\n\r\n\r\n bool addAll(T** keys, int size, const int tid) {\r\n for (int i = 0; i < size; i++) add(*keys[i], tid);\r\n return true;\r\n }\r\n};\r\n\r\n#endif /* _ONE_FILE_LF_LINKED_LIST_SET_H_ */\r\n" }, { "path": "datastructures/linkedlists/OFWFLinkedListSet.hpp", "content": " /*\r\n * Copyright 2017-2018\r\n * Andreia Correia \r\n * Pedro Ramalhete \r\n * Pascal Felber \r\n * Nachshon Cohen \r\n *\r\n * This work is published under the MIT license. See LICENSE.txt\r\n */\r\n#ifndef _ONEFILE_WF_LINKED_LIST_SET_H_\r\n#define _ONEFILE_WF_LINKED_LIST_SET_H_\r\n\r\n#include \r\n#include \r\n#include \"stms/OneFileWF.hpp\"\r\n\r\n\r\n/**\r\n *

A Linked List Set for One-File STM (wait-Free)

\r\n */\r\ntemplate\r\nclass OFWFLinkedListSet : public ofwf::tmbase {\r\n\r\nprivate:\r\n struct Node : public ofwf::tmbase {\r\n T key {};\r\n ofwf::tmtype next {nullptr};\r\n Node(T key) : key{key} { }\r\n Node() {}\r\n };\r\n\r\n alignas(128) ofwf::tmtype head {nullptr};\r\n alignas(128) ofwf::tmtype tail {nullptr};\r\n\r\n\r\npublic:\r\n OFWFLinkedListSet(unsigned int maxThreads=0) {\r\n ofwf::updateTx([this] () {\r\n Node* lhead = ofwf::tmNew();\r\n Node* ltail = ofwf::tmNew();\r\n head = lhead;\r\n head->next = ltail;\r\n tail = ltail;\r\n });\r\n }\r\n\r\n\r\n ~OFWFLinkedListSet() {\r\n ofwf::updateTx([this] () {\r\n // Delete all the nodes in the list\r\n Node* prev = head;\r\n Node* node = prev->next;\r\n while (node != tail) {\r\n ofwf::tmDelete(prev);\r\n prev = node;\r\n node = node->next;\r\n }\r\n ofwf::tmDelete(prev);\r\n ofwf::tmDelete(tail.pload());\r\n });\r\n }\r\n\r\n\r\n static std::string className() { return ofwf::OneFileWF::className() + \"-LinkedListSet\"; }\r\n\r\n\r\n /*\r\n * Progress Condition: wait-free\r\n * Adds a node with a key, returns false if the key is already in the set\r\n */\r\n bool add(T key, const int tid=0) {\r\n return ofwf::updateTx([this,key] () {\r\n Node* newNode = ofwf::tmNew(key);\r\n Node* prev = head;\r\n Node* node = prev->next;\r\n Node* ltail = tail;\r\n while (true) {\r\n if (node == ltail) break;\r\n T nkey = node->key;\r\n if (key == nkey) {\r\n ofwf::tmDelete(newNode); // If the key was already in the set, free the node that was never used\r\n return false;\r\n }\r\n if (nkey < key) break;\r\n prev = node;\r\n node = node->next;\r\n }\r\n prev->next = newNode;\r\n newNode->next = node;\r\n return true;\r\n });\r\n }\r\n\r\n\r\n /*\r\n * Progress Condition: wait-free\r\n * Removes a node with an key, returns false if the key is not in the set\r\n */\r\n bool remove(T key, const int tid=0) {\r\n return ofwf::updateTx([this,key] () {\r\n Node* prev = head;\r\n Node* node = prev->next;\r\n Node* ltail = tail;\r\n while (true) {\r\n if (node == ltail) return false;\r\n T nkey = node->key;\r\n if (key == nkey) {\r\n prev->next = node->next;\r\n ofwf::tmDelete(node);\r\n return true;\r\n }\r\n if (nkey < key) return false;\r\n prev = node;\r\n node = node->next;\r\n }\r\n });\r\n }\r\n\r\n\r\n /*\r\n * Progress Condition: wait-free\r\n * Returns true if it finds a node with a matching key\r\n */\r\n bool contains(T key, const int tid=0) {\r\n return ofwf::readTx([this,key] () {\r\n Node* node = head->next;\r\n Node* ltail = tail;\r\n while (true) {\r\n if (node == ltail) return false;\r\n T nkey = node->key;\r\n if (key == nkey) return true;\r\n if (nkey < key) return false;\r\n node = node->next;\r\n }\r\n });\r\n }\r\n\r\n\r\n bool addAll(T** keys, int size, const int tid) {\r\n for (int i = 0; i < size; i++) add(*keys[i], tid);\r\n return true;\r\n }\r\n};\r\n\r\n#endif /* _ONE_FILE_WF_LINKED_LIST_SET_H_ */\r\n" }, { "path": "datastructures/linkedlists/STMLinkedListSet.hpp", "content": "/******************************************************************************\r\n * Copyright (c) 2014-2018, Pedro Ramalhete, Andreia Correia\r\n * All rights reserved.\r\n *\r\n * Redistribution and use in source and binary forms, with or without\r\n * modification, are permitted provided that the following conditions are met:\r\n * * Redistributions of source code must retain the above copyright\r\n * notice, this list of conditions and the following disclaimer.\r\n * * Redistributions in binary form must reproduce the above copyright\r\n * notice, this list of conditions and the following disclaimer in the\r\n * documentation and/or other materials provided with the distribution.\r\n * * Neither the name of Concurrency Freaks nor the\r\n * names of its contributors may be used to endorse or promote products\r\n * derived from this software without specific prior written permission.\r\n *\r\n * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\r\n * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED\r\n * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\r\n * DISCLAIMED. IN NO EVENT SHALL BE LIABLE FOR ANY\r\n * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\r\n * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\r\n * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND\r\n * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\r\n * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\r\n * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\r\n ******************************************************************************\r\n */\r\n\r\n#ifndef _STM_LINKED_LIST_SET_H_\r\n#define _STM_LINKED_LIST_SET_H_\r\n\r\n#include \r\n#include \r\n//#include \"stms/OneFileLF.hpp\"\r\n\r\n\r\n/**\r\n *

A Linked List Set for an STM

\r\n *\r\n * TODO\r\n *\r\n *\r\n * @author Pedro Ramalhete\r\n * @author Andreia Correia\r\n */\r\ntemplate class TMTYPE, template class TMBASE>\r\nclass STMLinkedListSet : public TMBASE {\r\n\r\nprivate:\r\n struct Node : public TMBASE {\r\n T* key;\r\n TMTYPE next;\r\n Node(T* key) : key{key}, next{nullptr} { }\r\n };\r\n\r\n alignas(128) TMTYPE head {nullptr};\r\n alignas(128) TMTYPE tail {nullptr};\r\n\r\n\r\npublic:\r\n STMLinkedListSet(unsigned int maxThreads=0) {\r\n Node* lhead = new Node(nullptr);\r\n Node* ltail = new Node(nullptr);\r\n head = lhead;\r\n head->next = ltail;\r\n tail = ltail;\r\n }\r\n\r\n\r\n ~STMLinkedListSet() {\r\n // Delete all the nodes in the list\r\n Node* prev = head;\r\n Node* node = prev->next;\r\n while (node != tail) {\r\n delete prev;\r\n prev = node;\r\n node = node->next;\r\n }\r\n delete prev;\r\n delete tail;\r\n }\r\n\r\n\r\n static std::string className() { return TM::className() + \"-LinkedListSet\"; }\r\n\r\n\r\n /*\r\n * Progress Condition: lock-free\r\n * Adds a node with a key, returns false if the key is already in the set\r\n */\r\n bool add(T* key, const int tid=0) {\r\n if (key == nullptr) throw std::invalid_argument(\"key can not be nullptr\");\r\n return TM::updateTx([this,key] () -> bool {\r\n Node* newNode = TM::tmNew(key);\r\n Node* prev = head;\r\n Node* node = prev->next;\r\n while (true) {\r\n if (node == tail) break;\r\n if (*key == *node->key) {\r\n TM::tmDelete(newNode); // If the key was already in the set, free the node that was never used\r\n return false;\r\n }\r\n if (*(node->key) < *key) break;\r\n prev = node;\r\n node = node->next;\r\n }\r\n prev->next = newNode;\r\n newNode->next = node;\r\n return true;\r\n });\r\n }\r\n\r\n\r\n /*\r\n * Progress Condition: lock-free\r\n * Removes a node with an key, returns false if the key is not in the set\r\n */\r\n bool remove(T* key, const int tid=0) {\r\n if (key == nullptr) throw std::invalid_argument(\"key can not be nullptr\");\r\n return TM::updateTx([this,key] () -> bool {\r\n Node* prev = head;\r\n Node* node = prev->next;\r\n while (true) {\r\n if (node == tail) return false;\r\n if (*key == *node->key) {\r\n prev->next = node->next;\r\n TM::tmDelete(node);\r\n return true;\r\n }\r\n if (*(node->key) < *key) return false;\r\n prev = node;\r\n node = node->next;\r\n }\r\n });\r\n }\r\n\r\n\r\n /*\r\n * Progress Condition: lock-free\r\n * Returns true if it finds a node with a matching key\r\n */\r\n bool contains(T* key, const int tid=0) {\r\n if (key == nullptr) throw std::invalid_argument(\"key can not be nullptr\");\r\n return TM::readTx([this,key] () -> bool {\r\n Node* node = head->next;\r\n while (true) {\r\n if (node == tail) return false;\r\n if (*key == *node->key) return true;\r\n if (*(node->key) < *key) return false;\r\n node = node->next;\r\n }\r\n });\r\n }\r\n\r\n\r\n bool addAll(T** keys, int size, const int tid) {\r\n for (int i = 0; i < size; i++) add(keys[i], tid);\r\n }\r\n};\r\n\r\n#endif /* _ONE_FILE_LF_LINKED_LIST_SET_H_ */\r\n" }, { "path": "datastructures/linkedlists/TinySTMLinkedListSet.hpp", "content": "/*\r\n * Copyright 2017-2018\r\n * Andreia Correia \r\n * Pedro Ramalhete \r\n * Pascal Felber \r\n * Nachshon Cohen \r\n *\r\n * This work is published under the MIT license. See LICENSE.txt\r\n */\r\n#ifndef _TINY_STM_LINKED_LIST_SET_H_\r\n#define _TINY_STM_LINKED_LIST_SET_H_\r\n\r\n#include \"stms/TinySTM.hpp\"\r\n\r\n\r\n/**\r\n *

A Linked List Set for usage with TinySTM

\r\n */\r\ntemplate\r\nclass TinySTMLinkedListSet : public tinystm::tmbase {\r\n\r\nprivate:\r\n struct Node : public tinystm::tmbase {\r\n T key;\r\n tinystm::tmtype next{nullptr};\r\n Node() {}\r\n Node(T key) : key{key} { }\r\n };\r\n\r\n alignas(128) tinystm::tmtype head {nullptr};\r\n alignas(128) tinystm::tmtype tail {nullptr};\r\n\r\n\r\npublic:\r\n TinySTMLinkedListSet(unsigned int maxThreads=0) {\r\n tinystm::updateTx([this] () {\r\n Node* lhead = tinystm::tmNew();\r\n Node* ltail = tinystm::tmNew();\r\n head = lhead;\r\n head->next = ltail;\r\n tail = ltail;\r\n return true;\r\n });\r\n }\r\n\r\n\r\n ~TinySTMLinkedListSet() {\r\n tinystm::updateTx([this] () {\r\n // Delete all the nodes in the list\r\n Node* prev = head;\r\n Node* node = prev->next;\r\n while (node != tail) {\r\n tinystm::tmDelete(prev);\r\n prev = node;\r\n node = node->next;\r\n }\r\n tinystm::tmDelete(prev);\r\n tinystm::tmDelete(tail.load());\r\n return true;\r\n });\r\n }\r\n\r\n\r\n static std::string className() { return \"TinySTM-LinkedListSet\"; }\r\n\r\n\r\n /*\r\n * Progress Condition: blocking\r\n * Adds a node with a key, returns false if the key is already in the set\r\n */\r\n bool add(T key, const int tid=0) {\r\n return tinystm::updateTx([this,key] () {\r\n Node* newNode = tinystm::tmNew(key);\r\n Node* prev = head;\r\n Node* node = prev->next;\r\n Node* ltail = tail;\r\n while (true) {\r\n if (node == ltail) break;\r\n T nkey = node->key;\r\n if (key == nkey) {\r\n tinystm::tmDelete(newNode); // If the key was already in the set, free the node that was never used\r\n return false;\r\n }\r\n if (nkey < key) break;\r\n prev = node;\r\n node = node->next;\r\n }\r\n prev->next = newNode;\r\n newNode->next = node;\r\n return true;\r\n });\r\n }\r\n\r\n\r\n /*\r\n * Progress Condition: blocking\r\n * Removes a node with an key, returns false if the key is not in the set\r\n */\r\n bool remove(T key, const int tid=0) {\r\n return tinystm::updateTx([this,key] () {\r\n Node* prev = head;\r\n Node* node = prev->next;\r\n Node* ltail = tail;\r\n while (true) {\r\n if (node == ltail) return false;\r\n T nkey = node->key;\r\n if (key == nkey) {\r\n prev->next = node->next;\r\n tinystm::tmDelete(node);\r\n return true;\r\n }\r\n if (nkey < key) return false;\r\n prev = node;\r\n node = node->next;\r\n }\r\n });\r\n }\r\n\r\n\r\n /*\r\n * Progress Condition: blocking\r\n * Returns true if it finds a node with a matching key\r\n */\r\n bool contains(T key, const int tid=0) {\r\n return tinystm::readTx([this,key] () {\r\n Node* node = head->next;\r\n Node* ltail = tail;\r\n while (true) {\r\n if (node == ltail) return false;\r\n T nkey = node->key;\r\n if (key == nkey) return true;\r\n if (nkey < key) return false;\r\n node = node->next;\r\n }\r\n });\r\n }\r\n\r\n\r\n bool addAll(T** keys, int size, const int tid) {\r\n for (int i = 0; i < size; i++) add(*keys[i], tid);\r\n }\r\n};\r\n\r\n#endif /* _TINY_STM_LINKED_LIST_SET_H_ */\r\n" }, { "path": "datastructures/queues/CRWWPLinkedListQueue.hpp", "content": "/******************************************************************************\r\n * Copyright (c) 2014-2017, Pedro Ramalhete, Andreia Correia\r\n * All rights reserved.\r\n *\r\n * Redistribution and use in source and binary forms, with or without\r\n * modification, are permitted provided that the following conditions are met:\r\n * * Redistributions of source code must retain the above copyright\r\n * notice, this list of conditions and the following disclaimer.\r\n * * Redistributions in binary form must reproduce the above copyright\r\n * notice, this list of conditions and the following disclaimer in the\r\n * documentation and/or other materials provided with the distribution.\r\n * * Neither the name of Concurrency Freaks nor the\r\n * names of its contributors may be used to endorse or promote products\r\n * derived from this software without specific prior written permission.\r\n *\r\n * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\r\n * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED\r\n * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\r\n * DISCLAIMED. IN NO EVENT SHALL BE LIABLE FOR ANY\r\n * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\r\n * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\r\n * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND\r\n * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\r\n * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\r\n * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\r\n ******************************************************************************\r\n */\r\n\r\n#ifndef _CRWWP_LINKED_LIST_QUEUE_H_\r\n#define _CRWWP_LINKED_LIST_QUEUE_H_\r\n\r\n#include \r\n#include \r\n\r\n#include \"../../stms/CRWWPSTM.hpp\"\r\n\r\n/**\r\n *

A Linked List queue using C-RW-WP STM

\r\n *\r\n *\r\n * TODO\r\n *\r\n *\r\n * enqueue algorithm: sequential implementation + MWC\r\n * dequeue algorithm: sequential implementation + MWC\r\n * Consistency: Linearizable\r\n * enqueue() progress: lock-free\r\n * dequeue() progress: lock-free\r\n * Memory Reclamation: Hazard Eras (integrated into MWC)\r\n * enqueue min ops: 2 DCAS + 1 CAS\r\n * dequeue min ops: 1 DCAS + 1 CAS\r\n *\r\n * @author Pedro Ramalhete\r\n * @author Andreia Correia\r\n */\r\ntemplate\r\nclass CRWWPLinkedListQueue {\r\n\r\nprivate:\r\n struct Node : crwwpstm::tmbase {\r\n T* item;\r\n crwwpstm::tmtype next;\r\n Node(T* userItem) : item{userItem}, next{nullptr} { }\r\n };\r\n\r\n alignas(128) crwwpstm::tmtype head {nullptr};\r\n alignas(128) crwwpstm::tmtype tail {nullptr};\r\n\r\n\r\npublic:\r\n CRWWPLinkedListQueue(unsigned int maxThreads=0) {\r\n Node* sentinelNode = new Node(nullptr);\r\n head = sentinelNode;\r\n tail = sentinelNode;\r\n }\r\n\r\n\r\n ~CRWWPLinkedListQueue() {\r\n while (dequeue() != nullptr); // Drain the queue\r\n Node* lhead = head;\r\n delete lhead;\r\n }\r\n\r\n\r\n static std::string className() { return \"CRWWP-LinkedListQueue\"; }\r\n\r\n\r\n /*\r\n * Progress Condition: lock-free\r\n * Always returns true\r\n */\r\n bool enqueue(T* item, const int tid=0) {\r\n if (item == nullptr) throw std::invalid_argument(\"item can not be nullptr\");\r\n Node* newNode = crwwpstm::tmNew(item); // Let's allocate outside the transaction, less overhead\r\n return crwwpstm::updateTx([this,&newNode] () -> bool {\r\n tail->next = newNode;\r\n tail = newNode;\r\n return true;\r\n });\r\n }\r\n\r\n\r\n /*\r\n * Progress Condition: lock-free\r\n */\r\n T* dequeue(const int tid=0) {\r\n return crwwpstm::updateTx([this] () -> T* {\r\n Node* lhead = head;\r\n if (lhead == tail) return nullptr;\r\n head = lhead->next;\r\n crwwpstm::tmDelete(lhead);\r\n return head->item;\r\n });\r\n }\r\n};\r\n\r\n#endif /* _CRWWP_TM_LINKED_LIST_QUEUE_H_ */\r\n" }, { "path": "datastructures/queues/ESTMArrayLinkedListQueue.hpp", "content": "/******************************************************************************\r\n * Copyright (c) 2014-2017, Pedro Ramalhete, Andreia Correia\r\n * All rights reserved.\r\n *\r\n * Redistribution and use in source and binary forms, with or without\r\n * modification, are permitted provided that the following conditions are met:\r\n * * Redistributions of source code must retain the above copyright\r\n * notice, this list of conditions and the following disclaimer.\r\n * * Redistributions in binary form must reproduce the above copyright\r\n * notice, this list of conditions and the following disclaimer in the\r\n * documentation and/or other materials provided with the distribution.\r\n * * Neither the name of Concurrency Freaks nor the\r\n * names of its contributors may be used to endorse or promote products\r\n * derived from this software without specific prior written permission.\r\n *\r\n * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\r\n * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED\r\n * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\r\n * DISCLAIMED. IN NO EVENT SHALL BE LIABLE FOR ANY\r\n * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\r\n * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\r\n * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND\r\n * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\r\n * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\r\n * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\r\n ******************************************************************************\r\n */\r\n\r\n#ifndef _ELASTIC_STM_ARRAY_LINKED_LIST_QUEUE_H_\r\n#define _ELASTIC_STM_ARRAY_LINKED_LIST_QUEUE_H_\r\n\r\n#include \r\n#include \r\n\r\n#include \"stms/ESTM.hpp\"\r\n\r\n/**\r\n *

An Array Linked List Queue using OneFile STM (Wait-Free)

\r\n *\r\n * TODO\r\n *\r\n *\r\n * enqueue algorithm: sequential implementation + MWC\r\n * dequeue algorithm: sequential implementation + MWC\r\n * Consistency: Linearizable\r\n * enqueue() progress: lock-free\r\n * dequeue() progress: lock-free\r\n * Memory Reclamation: Hazard Eras (integrated into MWC)\r\n * enqueue min ops: 2 DCAS + 1 CAS\r\n * dequeue min ops: 1 DCAS + 1 CAS\r\n *\r\n * @author Pedro Ramalhete\r\n * @author Andreia Correia\r\n */\r\ntemplate\r\nclass ESTMArrayLinkedListQueue {\r\n\r\nprivate:\r\n struct Node : estm::tmbase {\r\n static const int ITEM_NUM = 1024;\r\n estm::tmtype headidx {0};\r\n estm::tmtype items[ITEM_NUM];\r\n estm::tmtype tailidx {0};\r\n estm::tmtype next {nullptr};\r\n Node(T* item) {\r\n items[0] = item;\r\n tailidx = 1;\r\n headidx = 0;\r\n for (int i = 1; i < ITEM_NUM; i++) items[i] = nullptr;\r\n }\r\n };\r\n\r\n alignas(128) estm::tmtype head {nullptr};\r\n alignas(128) estm::tmtype tail {nullptr};\r\n\r\n\r\npublic:\r\n ESTMArrayLinkedListQueue(unsigned int maxThreads=0) {\r\n estm::updateTx([this] () {\r\n Node* sentinelNode = estm::tmNew(nullptr);\r\n sentinelNode->tailidx = 0;\r\n head = sentinelNode;\r\n tail = sentinelNode;\r\n return true;\r\n });\r\n }\r\n\r\n\r\n ~ESTMArrayLinkedListQueue() {\r\n while (dequeue() != nullptr); // Drain the queue\r\n estm::updateTx([this] () {\r\n Node* lhead = head;\r\n estm::tmDelete(lhead);\r\n return true;\r\n });\r\n }\r\n\r\n\r\n static std::string className() { return \"ESTM-ArrayLinkedListQueue\"; }\r\n\r\n\r\n /*\r\n * Progress Condition: blocking\r\n * Always returns true\r\n */\r\n bool enqueue(T* item, const int tid=0) {\r\n if (item == nullptr) throw std::invalid_argument(\"item can not be nullptr\");\r\n return estm::updateTx([this,item] () -> bool {\r\n Node* ltail = tail;\r\n uint64_t ltailidx = ltail->tailidx;\r\n if (ltailidx < Node::ITEM_NUM) {\r\n ltail->items[ltailidx] = item;\r\n ++ltail->tailidx;\r\n return true;\r\n }\r\n Node* newNode = estm::tmNew(item);\r\n tail->next = newNode;\r\n tail = newNode;\r\n return true;\r\n });\r\n }\r\n\r\n\r\n /*\r\n * Progress Condition: blocking\r\n */\r\n T* dequeue(const int tid=0) {\r\n return estm::updateTx([this] () -> T* {\r\n Node* lhead = head;\r\n uint64_t lheadidx = lhead->headidx;\r\n // Check if queue is empty\r\n if (lhead == tail && lheadidx == tail->tailidx) return nullptr;\r\n if (lheadidx < Node::ITEM_NUM) {\r\n ++lhead->headidx;\r\n return lhead->items[lheadidx];\r\n }\r\n lhead = lhead->next;\r\n estm::tmDelete(head.load());\r\n head = lhead;\r\n ++lhead->headidx;\r\n return lhead->items[0];\r\n });\r\n }\r\n};\r\n\r\n#endif /* _OF_WF_ARRAY_LINKED_LIST_QUEUE_H_ */\r\n" }, { "path": "datastructures/queues/ESTMLinkedListQueue.hpp", "content": "/******************************************************************************\r\n * Copyright (c) 2014-2018, Pedro Ramalhete, Andreia Correia\r\n * All rights reserved.\r\n *\r\n * Redistribution and use in source and binary forms, with or without\r\n * modification, are permitted provided that the following conditions are met:\r\n * * Redistributions of source code must retain the above copyright\r\n * notice, this list of conditions and the following disclaimer.\r\n * * Redistributions in binary form must reproduce the above copyright\r\n * notice, this list of conditions and the following disclaimer in the\r\n * documentation and/or other materials provided with the distribution.\r\n * * Neither the name of Concurrency Freaks nor the\r\n * names of its contributors may be used to endorse or promote products\r\n * derived from this software without specific prior written permission.\r\n *\r\n * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\r\n * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED\r\n * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\r\n * DISCLAIMED. IN NO EVENT SHALL BE LIABLE FOR ANY\r\n * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\r\n * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\r\n * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND\r\n * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\r\n * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\r\n * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\r\n ******************************************************************************\r\n */\r\n\r\n#ifndef _ELASTIC_STM_LINKED_LIST_QUEUE_H_\r\n#define _ELASTIC_STM_LINKED_LIST_QUEUE_H_\r\n\r\n#include \r\n#include \r\n\r\n#include \"stms/ESTM.hpp\"\r\n\r\n/**\r\n *

A Linked List queue using Elastic STM (blocking)

\r\n *\r\n *\r\n * TODO\r\n *\r\n * @author Pedro Ramalhete\r\n * @author Andreia Correia\r\n */\r\ntemplate\r\nclass ESTMLinkedListQueue {\r\n\r\nprivate:\r\n struct Node : estm::tmbase {\r\n T* item;\r\n estm::tmtype next {nullptr};\r\n Node(T* userItem) : item{userItem} { }\r\n };\r\n\r\n alignas(128) estm::tmtype head {nullptr};\r\n alignas(128) estm::tmtype tail {nullptr};\r\n\r\n\r\npublic:\r\n ESTMLinkedListQueue(unsigned int maxThreads=0) {\r\n estm::updateTx([this] () {\r\n Node* sentinelNode = estm::tmNew(nullptr);\r\n head = sentinelNode;\r\n tail = sentinelNode;\r\n return true;\r\n });\r\n }\r\n\r\n\r\n ~ESTMLinkedListQueue() {\r\n while (dequeue() != nullptr); // Drain the queue\r\n estm::updateTx([this] () {\r\n Node* lhead = head;\r\n estm::tmDelete(lhead);\r\n return true;\r\n });\r\n }\r\n\r\n\r\n static std::string className() { return \"ESTM-LinkedListQueue\"; }\r\n\r\n\r\n /*\r\n * Progress Condition: blocking\r\n * Always returns true\r\n */\r\n bool enqueue(T* item, const int tid=0) {\r\n if (item == nullptr) throw std::invalid_argument(\"item can not be nullptr\");\r\n return estm::updateTx([this,item] () -> bool {\r\n Node* newNode = estm::tmNew(item);\r\n tail->next = newNode;\r\n tail = newNode;\r\n return true;\r\n });\r\n }\r\n\r\n\r\n /*\r\n * Progress Condition: blocking\r\n */\r\n T* dequeue(const int tid=0) {\r\n return estm::updateTx([this] () -> T* {\r\n Node* lhead = head;\r\n if (lhead == tail) return nullptr;\r\n head = lhead->next;\r\n estm::tmDelete(lhead);\r\n return head->item;\r\n });\r\n }\r\n};\r\n\r\n#endif /* _ESTM_LINKED_LIST_QUEUE_H_ */\r\n" }, { "path": "datastructures/queues/FAAArrayQueue.hpp", "content": "/******************************************************************************\r\n * Copyright (c) 2014-2016, Pedro Ramalhete, Andreia Correia\r\n * All rights reserved.\r\n *\r\n * Redistribution and use in source and binary forms, with or without\r\n * modification, are permitted provided that the following conditions are met:\r\n * * Redistributions of source code must retain the above copyright\r\n * notice, this list of conditions and the following disclaimer.\r\n * * Redistributions in binary form must reproduce the above copyright\r\n * notice, this list of conditions and the following disclaimer in the\r\n * documentation and/or other materials provided with the distribution.\r\n * * Neither the name of Concurrency Freaks nor the\r\n * names of its contributors may be used to endorse or promote products\r\n * derived from this software without specific prior written permission.\r\n *\r\n * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\r\n * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED\r\n * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\r\n * DISCLAIMED. IN NO EVENT SHALL BE LIABLE FOR ANY\r\n * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\r\n * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\r\n * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND\r\n * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\r\n * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\r\n * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\r\n ******************************************************************************\r\n */\r\n\r\n#ifndef _FAA_ARRAY_QUEUE_HP_H_\r\n#define _FAA_ARRAY_QUEUE_HP_H_\r\n\r\n#include \r\n#include \r\n#include \"HazardPointers.hpp\"\r\n\r\n\r\n/**\r\n *

Fetch-And-Add Array Queue

\r\n *\r\n * Each node has one array but we don't search for a vacant entry. Instead, we\r\n * use FAA to obtain an index in the array, for enqueueing or dequeuing.\r\n *\r\n * There are some similarities between this queue and the basic queue in YMC:\r\n * http://chaoran.me/assets/pdf/wfq-ppopp16.pdf\r\n * but it's not the same because the queue in listing 1 is obstruction-free, while\r\n * our algorithm is lock-free.\r\n * In FAAArrayQueue eventually a new node will be inserted (using Michael-Scott's\r\n * algorithm) and it will have an item pre-filled in the first position, which means\r\n * that at most, after BUFFER_SIZE steps, one item will be enqueued (and it can then\r\n * be dequeued). This kind of progress is lock-free.\r\n *\r\n * Each entry in the array may contain one of three possible values:\r\n * - A valid item that has been enqueued;\r\n * - nullptr, which means no item has yet been enqueued in that position;\r\n * - taken, a special value that means there was an item but it has been dequeued;\r\n *\r\n * Enqueue algorithm: FAA + CAS(null,item)\r\n * Dequeue algorithm: FAA + CAS(item,taken)\r\n * Consistency: Linearizable\r\n * enqueue() progress: lock-free\r\n * dequeue() progress: lock-free\r\n * Memory Reclamation: Hazard Pointers (lock-free)\r\n * Uncontended enqueue: 1 FAA + 1 CAS + 1 HP\r\n * Uncontended dequeue: 1 FAA + 1 CAS + 1 HP\r\n *\r\n *\r\n *

\r\n * Lock-Free Linked List as described in Maged Michael and Michael Scott's paper:\r\n * {@link http://www.cs.rochester.edu/~scott/papers/1996_PODC_queues.pdf}\r\n * \r\n * Simple, Fast, and Practical Non-Blocking and Blocking Concurrent Queue Algorithms\r\n *

\r\n * The paper on Hazard Pointers is named \"Hazard Pointers: Safe Memory\r\n * Reclamation for Lock-Free objects\" and it is available here:\r\n * http://web.cecs.pdx.edu/~walpole/class/cs510/papers/11.pdf\r\n *\r\n * @author Pedro Ramalhete\r\n * @author Andreia Correia\r\n */\r\ntemplate\r\nclass FAAArrayQueue {\r\n static const long BUFFER_SIZE = 1024; // 1024\r\n\r\nprivate:\r\n struct Node {\r\n alignas(128) std::atomic deqidx;\r\n alignas(128) std::atomic items[BUFFER_SIZE];\r\n alignas(128) std::atomic enqidx;\r\n alignas(128) std::atomic next;\r\n\r\n // Start with the first entry pre-filled and enqidx at 1\r\n Node(T* item) : deqidx{0}, enqidx{1}, next{nullptr} {\r\n items[0].store(item, std::memory_order_relaxed);\r\n for (long i = 1; i < BUFFER_SIZE; i++) {\r\n items[i].store(nullptr, std::memory_order_relaxed);\r\n }\r\n }\r\n\r\n bool casNext(Node *cmp, Node *val) {\r\n return next.compare_exchange_strong(cmp, val);\r\n }\r\n };\r\n\r\n bool casTail(Node *cmp, Node *val) {\r\n\t\treturn tail.compare_exchange_strong(cmp, val);\r\n\t}\r\n\r\n bool casHead(Node *cmp, Node *val) {\r\n return head.compare_exchange_strong(cmp, val);\r\n }\r\n\r\n // Pointers to head and tail of the list\r\n alignas(128) std::atomic head;\r\n alignas(128) std::atomic tail;\r\n\r\n static const int MAX_THREADS = 128;\r\n const int maxThreads;\r\n\r\n T* taken = (T*)new int(); // Muuuahahah !\r\n\r\n // We need just one hazard pointer\r\n HazardPointers hp {1, maxThreads};\r\n const int kHpTail = 0;\r\n const int kHpHead = 0;\r\n\r\n\r\npublic:\r\n FAAArrayQueue(int maxThreads=MAX_THREADS) : maxThreads{maxThreads} {\r\n Node* sentinelNode = new Node(nullptr);\r\n sentinelNode->enqidx.store(0, std::memory_order_relaxed);\r\n head.store(sentinelNode, std::memory_order_relaxed);\r\n tail.store(sentinelNode, std::memory_order_relaxed);\r\n }\r\n\r\n\r\n ~FAAArrayQueue() {\r\n while (dequeue(0) != nullptr); // Drain the queue\r\n delete head.load(); // Delete the last node\r\n delete (int*)taken;\r\n }\r\n\r\n\r\n static std::string className() { return \"FAAArrayQueue\"; }\r\n\r\n\r\n void enqueue(T* item, const int tid) {\r\n if (item == nullptr) throw std::invalid_argument(\"item can not be nullptr\");\r\n while (true) {\r\n Node* ltail = hp.protect(kHpTail, tail, tid);\r\n const int idx = ltail->enqidx.fetch_add(1);\r\n if (idx > BUFFER_SIZE-1) { // This node is full\r\n if (ltail != tail.load()) continue;\r\n Node* lnext = ltail->next.load();\r\n if (lnext == nullptr) {\r\n Node* newNode = new Node(item);\r\n if (ltail->casNext(nullptr, newNode)) {\r\n casTail(ltail, newNode);\r\n hp.clear(tid);\r\n return;\r\n }\r\n delete newNode;\r\n } else {\r\n casTail(ltail, lnext);\r\n }\r\n continue;\r\n }\r\n T* itemnull = nullptr;\r\n if (ltail->items[idx].compare_exchange_strong(itemnull, item)) {\r\n hp.clear(tid);\r\n return;\r\n }\r\n }\r\n }\r\n\r\n\r\n T* dequeue(const int tid) {\r\n while (true) {\r\n Node* lhead = hp.protect(kHpHead, head, tid);\r\n if (lhead->deqidx.load() >= lhead->enqidx.load() && lhead->next.load() == nullptr) break;\r\n const int idx = lhead->deqidx.fetch_add(1);\r\n if (idx > BUFFER_SIZE-1) { // This node has been drained, check if there is another one\r\n Node* lnext = lhead->next.load();\r\n if (lnext == nullptr) break; // No more nodes in the queue\r\n if (casHead(lhead, lnext)) hp.retire(lhead, tid);\r\n continue;\r\n }\r\n T* item = lhead->items[idx].load();\r\n if (item != nullptr) {\r\n hp.clear(tid);\r\n return item;\r\n }\r\n item = lhead->items[idx].exchange(taken);\r\n if (item == nullptr) continue;\r\n hp.clear(tid);\r\n return item;\r\n }\r\n hp.clear(tid);\r\n return nullptr;\r\n }\r\n};\r\n\r\n#endif /* _FAA_ARRAY_QUEUE_HP_H_ */\r\n" }, { "path": "datastructures/queues/HazardPointers.hpp", "content": "/******************************************************************************\r\n * Copyright (c) 2014-2016, Pedro Ramalhete, Andreia Correia\r\n * All rights reserved.\r\n *\r\n * Redistribution and use in source and binary forms, with or without\r\n * modification, are permitted provided that the following conditions are met:\r\n * * Redistributions of source code must retain the above copyright\r\n * notice, this list of conditions and the following disclaimer.\r\n * * Redistributions in binary form must reproduce the above copyright\r\n * notice, this list of conditions and the following disclaimer in the\r\n * documentation and/or other materials provided with the distribution.\r\n * * Neither the name of Concurrency Freaks nor the\r\n * names of its contributors may be used to endorse or promote products\r\n * derived from this software without specific prior written permission.\r\n\r\n *\r\n * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\r\n * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED\r\n * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\r\n * DISCLAIMED. IN NO EVENT SHALL BE LIABLE FOR ANY\r\n * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\r\n * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\r\n * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND\r\n * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\r\n * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\r\n * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\r\n ******************************************************************************\r\n */\r\n\r\n#ifndef _HAZARD_POINTERS_H_\r\n#define _HAZARD_POINTERS_H_\r\n\r\n#include \r\n#include \r\n#include \r\n#include \r\n\r\n\r\ntemplate\r\nclass HazardPointers {\r\n\r\nprivate:\r\n static const int HP_MAX_THREADS = 128;\r\n static const int HP_MAX_HPS = 128; // This is named 'K' in the HP paper\r\n static const int CLPAD = 128/sizeof(std::atomic);\r\n static const int HP_THRESHOLD_R = 0; // This is named 'R' in the HP paper\r\n static const int MAX_RETIRED = HP_MAX_THREADS*HP_MAX_HPS; // Maximum number of retired objects per thread\r\n\r\n const int maxHPs;\r\n const int maxThreads;\r\n\r\n alignas(128) std::atomic* hp[HP_MAX_THREADS];\r\n // It's not nice that we have a lot of empty vectors, but we need padding to avoid false sharing\r\n alignas(128) std::vector retiredList[HP_MAX_THREADS*CLPAD];\r\n std::function defdeleter = [](T* t, int tid){ delete t; };\r\n std::function& deleter;\r\npublic:\r\n\r\n HazardPointers(int maxHPs, int maxThreads) : maxHPs{maxHPs}, maxThreads{maxThreads}, deleter{defdeleter} {\r\n for (int ithread = 0; ithread < HP_MAX_THREADS; ithread++) {\r\n hp[ithread] = new std::atomic[HP_MAX_HPS];\r\n for (int ihp = 0; ihp < HP_MAX_HPS; ihp++) {\r\n hp[ithread][ihp].store(nullptr, std::memory_order_relaxed);\r\n }\r\n }\r\n }\r\n\r\n HazardPointers(int maxHPs, int maxThreads, std::function& deleter) : maxHPs{maxHPs}, maxThreads{maxThreads}, deleter{deleter} {\r\n for (int ithread = 0; ithread < HP_MAX_THREADS; ithread++) {\r\n hp[ithread] = new std::atomic[HP_MAX_HPS];\r\n for (int ihp = 0; ihp < HP_MAX_HPS; ihp++) {\r\n hp[ithread][ihp].store(nullptr, std::memory_order_relaxed);\r\n }\r\n }\r\n }\r\n\r\n ~HazardPointers() {\r\n for (int ithread = 0; ithread < HP_MAX_THREADS; ithread++) {\r\n delete[] hp[ithread];\r\n // Clear the current retired nodes\r\n for (unsigned iret = 0; iret < retiredList[ithread*CLPAD].size(); iret++) {\r\n delete retiredList[ithread*CLPAD][iret];\r\n }\r\n }\r\n }\r\n\r\n\r\n /**\r\n * Progress Condition: wait-free bounded (by maxHPs)\r\n */\r\n void clear(const int tid) {\r\n for (int ihp = 0; ihp < maxHPs; ihp++) {\r\n hp[tid][ihp].store(nullptr, std::memory_order_release);\r\n }\r\n }\r\n\r\n\r\n /**\r\n * Progress Condition: wait-free population oblivious\r\n */\r\n void clearOne(int ihp, const int tid) {\r\n hp[tid][ihp].store(nullptr, std::memory_order_release);\r\n }\r\n\r\n\r\n /**\r\n * Progress Condition: lock-free\r\n */\r\n T* protect(int index, const std::atomic& atom, const int tid) {\r\n T* n = nullptr;\r\n T* ret;\r\n\t\twhile ((ret = atom.load()) != n) {\r\n\t\t\thp[tid][index].store(ret);\r\n\t\t\tn = ret;\r\n\t\t}\r\n\t\treturn ret;\r\n }\r\n\r\n T* get(int index, const int tid){\r\n return hp[tid][index].load();\r\n }\r\n /**\r\n * This returns the same value that is passed as ptr, which is sometimes useful\r\n * Progress Condition: wait-free population oblivious\r\n */\r\n T* protectPtr(int index, T* ptr, const int tid) {\r\n hp[tid][index].store(ptr);\r\n return ptr;\r\n }\r\n\r\n\r\n\r\n /**\r\n * This returns the same value that is passed as ptr, which is sometimes useful\r\n * Progress Condition: wait-free population oblivious\r\n */\r\n T* protectPtrRelease(int index, T* ptr, const int tid) {\r\n hp[tid][index].store(ptr, std::memory_order_release);\r\n return ptr;\r\n }\r\n\r\n\r\n /**\r\n * Progress Condition: wait-free bounded (by the number of threads squared)\r\n */\r\n void retire(T* ptr, const int tid) {\r\n retiredList[tid*CLPAD].push_back(ptr);\r\n if (retiredList[tid*CLPAD].size() < HP_THRESHOLD_R) return;\r\n for (unsigned iret = 0; iret < retiredList[tid*CLPAD].size();) {\r\n auto obj = retiredList[tid*CLPAD][iret];\r\n bool canDelete = true;\r\n for (int tid = 0; tid < maxThreads && canDelete; tid++) {\r\n for (int ihp = maxHPs-1; ihp >= 0; ihp--) {\r\n if (hp[tid][ihp].load() == obj) {\r\n canDelete = false;\r\n break;\r\n }\r\n }\r\n }\r\n if (canDelete) {\r\n retiredList[tid*CLPAD].erase(retiredList[tid*CLPAD].begin() + iret);\r\n deleter(obj,tid);\r\n continue;\r\n }\r\n iret++;\r\n }\r\n }\r\n};\r\n\r\n#endif /* _HAZARD_POINTERS_H_ */\r\n" }, { "path": "datastructures/queues/HazardPointersSimQueue.hpp", "content": "/******************************************************************************\r\n * Copyright (c) 2014-2016, Pedro Ramalhete, Andreia Correia\r\n * All rights reserved.\r\n *\r\n * Redistribution and use in source and binary forms, with or without\r\n * modification, are permitted provided that the following conditions are met:\r\n * * Redistributions of source code must retain the above copyright\r\n * notice, this list of conditions and the following disclaimer.\r\n * * Redistributions in binary form must reproduce the above copyright\r\n * notice, this list of conditions and the following disclaimer in the\r\n * documentation and/or other materials provided with the distribution.\r\n * * Neither the name of Concurrency Freaks nor the\r\n * names of its contributors may be used to endorse or promote products\r\n * derived from this software without specific prior written permission.\r\n\r\n *\r\n * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\r\n * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED\r\n * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\r\n * DISCLAIMED. IN NO EVENT SHALL BE LIABLE FOR ANY\r\n * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\r\n * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\r\n * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND\r\n * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\r\n * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\r\n * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\r\n ******************************************************************************\r\n */\r\n\r\n#ifndef _HAZARD_POINTERS_SIM_QUEUE_H_\r\n#define _HAZARD_POINTERS_SIM_QUEUE_H_\r\n\r\n#include \r\n#include \r\n#include \r\n#include \r\n\r\n\r\n// TODO: use std::vector instead of arrays for the retired objects (keep the padding)\r\ntemplate\r\nclass HazardPointersSimQueue {\r\n\r\nprivate:\r\n static const int HP_MAX_THREADS = 128;\r\n static const int HP_MAX_HPS = 11; // This is named 'K' in the HP paper\r\n static const int CLPAD = 128/sizeof(std::atomic);\r\n static const int HP_THRESHOLD_R = 0; // This is named 'R' in the HP paper\r\n static const int MAX_RETIRED = HP_MAX_THREADS*HP_MAX_HPS; // Maximum number of retired objects per thread\r\n\r\n const int maxHPs;\r\n const int maxThreads;\r\n\r\n std::atomic hp[HP_MAX_THREADS*CLPAD][HP_MAX_HPS];\r\n // It's not nice that we have a lot of empty vectors, but we need padding to avoid false sharing\r\n std::vector retiredList[HP_MAX_THREADS*CLPAD];\r\n\r\n std::function findPtr;\r\n\r\npublic:\r\n HazardPointersSimQueue(std::function& find, int maxHPs=HP_MAX_HPS, int maxThreads=HP_MAX_THREADS) : maxHPs{maxHPs}, maxThreads{maxThreads} {\r\n findPtr = find;\r\n for (int ithread = 0; ithread < HP_MAX_THREADS; ithread++) {\r\n for (int ihp = 0; ihp < HP_MAX_HPS; ihp++) {\r\n hp[ithread*CLPAD][ihp].store(nullptr, std::memory_order_relaxed);\r\n }\r\n }\r\n }\r\n\r\n ~HazardPointersSimQueue() {\r\n for (int ithread = 0; ithread < HP_MAX_THREADS; ithread++) {\r\n // Clear the current retired nodes\r\n for (unsigned iret = 0; iret < retiredList[ithread*CLPAD].size(); iret++) {\r\n delete retiredList[ithread*CLPAD][iret];\r\n }\r\n }\r\n }\r\n\r\n\r\n /**\r\n * Progress Condition: wait-free bounded (by maxHPs)\r\n */\r\n void clear(const int tid) {\r\n for (int ihp = 0; ihp < maxHPs; ihp++) {\r\n hp[tid*CLPAD][ihp].store(nullptr, std::memory_order_release);\r\n }\r\n }\r\n\r\n\r\n /**\r\n * Progress Condition: wait-free population oblivious\r\n */\r\n void clearOne(int ihp, const int tid) {\r\n hp[tid*CLPAD][ihp].store(nullptr, std::memory_order_release);\r\n }\r\n\r\n\r\n /**\r\n * Progress Condition: lock-free\r\n */\r\n T* protect(int index, const std::atomic& atom, const int tid) {\r\n T* n = nullptr;\r\n T* ret;\r\n\t\twhile ((ret = atom.load()) != n) {\r\n\t\t\thp[tid*CLPAD][index].store(ret);\r\n\t\t\tn = ret;\r\n\t\t}\r\n\t\treturn ret;\r\n }\r\n\r\n /**\r\n * This returns the same value that is passed as ptr, which is sometimes useful\r\n * Progress Condition: wait-free population oblivious\r\n */\r\n T* protectPtr(int index, T* ptr, const int tid) {\r\n hp[tid*CLPAD][index].store(ptr);\r\n return ptr;\r\n }\r\n\r\n\r\n /**\r\n * This returns the same value that is passed as ptr, which is sometimes useful\r\n * Progress Condition: wait-free population oblivious\r\n */\r\n T* protectRelease(int index, T* ptr, const int tid) {\r\n hp[tid*CLPAD][index].store(ptr, std::memory_order_release);\r\n return ptr;\r\n }\r\n\r\n\r\n /**\r\n * This returns the same value that is passed as ptr, which is sometimes useful\r\n * Progress Condition: wait-free bounded (by the number of threads squared)\r\n */\r\n void retire(T* ptr, const int tid) {\r\n retiredList[tid*CLPAD].push_back(ptr);\r\n for (unsigned iret = 0; iret < retiredList[tid*CLPAD].size();) {\r\n auto obj = retiredList[tid*CLPAD][iret];\r\n if (findPtr(obj)) {\r\n iret++;\r\n continue;\r\n }\r\n bool canDelete = true;\r\n for (int tid = 0; tid < maxThreads && canDelete; tid++) {\r\n for (int ihp = maxHPs-1; ihp >= 0; ihp--) {\r\n if (hp[tid*CLPAD][ihp].load() == obj) {\r\n canDelete = false;\r\n break;\r\n }\r\n }\r\n }\r\n if (canDelete) {\r\n retiredList[tid*CLPAD].erase(retiredList[tid*CLPAD].begin() + iret);\r\n delete obj;\r\n continue;\r\n }\r\n iret++;\r\n }\r\n }\r\n};\r\n\r\n#endif /* _HAZARD_POINTERS_H_ */\r\n" }, { "path": "datastructures/queues/LCRQueue.hpp", "content": "/******************************************************************************\r\n * Copyright (c) 2014-2016, Pedro Ramalhete, Andreia Correia\r\n * All rights reserved.\r\n *\r\n * Redistribution and use in source and binary forms, with or without\r\n * modification, are permitted provided that the following conditions are met:\r\n * * Redistributions of source code must retain the above copyright\r\n * notice, this list of conditions and the following disclaimer.\r\n * * Redistributions in binary form must reproduce the above copyright\r\n * notice, this list of conditions and the following disclaimer in the\r\n * documentation and/or other materials provided with the distribution.\r\n * * Neither the name of Concurrency Freaks nor the\r\n * names of its contributors may be used to endorse or promote products\r\n * derived from this software without specific prior written permission.\r\n *\r\n * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\r\n * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED\r\n * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\r\n * DISCLAIMED. IN NO EVENT SHALL BE LIABLE FOR ANY\r\n * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\r\n * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\r\n * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND\r\n * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\r\n * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\r\n * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\r\n ******************************************************************************\r\n */\r\n\r\n#ifndef _LCRQ_QUEUE_HP_H_\r\n#define _LCRQ_QUEUE_HP_H_\r\n\r\n\r\n#include \r\n#include \"HazardPointers.hpp\"\r\n\r\n// CAS2 macro\r\n\r\n#define __CAS2(ptr, o1, o2, n1, n2) \\\r\n({ \\\r\n char __ret; \\\r\n __typeof__(o2) __junk; \\\r\n __typeof__(*(ptr)) __old1 = (o1); \\\r\n __typeof__(o2) __old2 = (o2); \\\r\n __typeof__(*(ptr)) __new1 = (n1); \\\r\n __typeof__(o2) __new2 = (n2); \\\r\n asm volatile(\"lock cmpxchg16b %2;setz %1\" \\\r\n : \"=d\"(__junk), \"=a\"(__ret), \"+m\" (*ptr) \\\r\n : \"b\"(__new1), \"c\"(__new2), \\\r\n \"a\"(__old1), \"d\"(__old2)); \\\r\n __ret; })\r\n\r\n#define CAS2(ptr, o1, o2, n1, n2) __CAS2(ptr, o1, o2, n1, n2)\r\n\r\n\r\n#define BIT_TEST_AND_SET(ptr, b) \\\r\n({ \\\r\n char __ret; \\\r\n asm volatile(\"lock btsq $63, %0; setnc %1\" : \"+m\"(*ptr), \"=a\"(__ret) : : \"cc\"); \\\r\n __ret; \\\r\n})\r\n\r\n\r\n\r\n\r\n/**\r\n *

LCRQ Queue

\r\n *\r\n * This is LCRQ by Adam Morrison and Yehuda Afek\r\n * http://www.cs.tau.ac.il/~mad/publications/ppopp2013-x86queues.pdf\r\n *\r\n * This implementation does NOT obey the C++ memory model rules AND it is x86 specific.\r\n * No guarantees are given on the correctness or consistency of the results if you use this queue.\r\n *\r\n * Bugs fixed:\r\n * tt was not initialized in dequeue();\r\n *\r\n *

\r\n * enqueue algorithm: MS enqueue + LCRQ with re-usage\r\n * dequeue algorithm: MS dequeue + LCRQ with re-usage\r\n * Consistency: Linearizable\r\n * enqueue() progress: lock-free\r\n * dequeue() progress: lock-free\r\n * Memory Reclamation: Hazard Pointers (lock-free)\r\n *\r\n *

\r\n * The paper on Hazard Pointers is named \"Hazard Pointers: Safe Memory\r\n * Reclamation for Lock-Free objects\" and it is available here:\r\n * http://web.cecs.pdx.edu/~walpole/class/cs510/papers/11.pdf\r\n *\r\n * @author Pedro Ramalhete\r\n * @author Andreia Correia\r\n */\r\ntemplate\r\nclass LCRQueue {\r\n\r\nprivate:\r\n static const int RING_POW = 10;\r\n static const uint64_t RING_SIZE = 1ull << RING_POW;\r\n\r\n struct Cell {\r\n std::atomic val;\r\n std::atomic idx;\r\n uint64_t pad[14];\r\n } __attribute__ ((aligned (128)));\r\n\r\n struct Node {\r\n std::atomic head __attribute__ ((aligned (128)));\r\n std::atomic tail __attribute__ ((aligned (128)));\r\n std::atomic next __attribute__ ((aligned (128)));\r\n Cell array[RING_SIZE];\r\n\r\n Node() {\r\n for (unsigned i = 0; i < RING_SIZE; i++) {\r\n array[i].val.store(nullptr, std::memory_order_relaxed);\r\n array[i].idx.store(i, std::memory_order_relaxed);\r\n }\r\n head.store(0, std::memory_order_relaxed);\r\n tail.store(0, std::memory_order_relaxed);\r\n next.store(nullptr, std::memory_order_relaxed);\r\n }\r\n };\r\n\r\n alignas(128) std::atomic head;\r\n alignas(128) std::atomic tail;\r\n\r\n static const int MAX_THREADS = 128;\r\n const int maxThreads;\r\n\r\n HazardPointers hp {1, maxThreads};\r\n const int kHpTail = 0;\r\n const int kHpHead = 0;\r\n\r\n\r\n /*\r\n * Private methods\r\n */\r\n int is_empty(T* v) {\r\n return (v == nullptr);\r\n }\r\n\r\n uint64_t node_index(uint64_t i) {\r\n return (i & ~(1ull << 63));\r\n }\r\n\r\n uint64_t set_unsafe(uint64_t i) {\r\n return (i | (1ull << 63));\r\n }\r\n\r\n uint64_t node_unsafe(uint64_t i) {\r\n return (i & (1ull << 63));\r\n }\r\n\r\n inline uint64_t tail_index(uint64_t t) {\r\n return (t & ~(1ull << 63));\r\n }\r\n\r\n int crq_is_closed(uint64_t t) {\r\n return (t & (1ull << 63)) != 0;\r\n }\r\n\r\n void fixState(Node *lhead) {\r\n while (1) {\r\n uint64_t t = lhead->tail.fetch_add(0);\r\n uint64_t h = lhead->head.fetch_add(0);\r\n // TODO: is it ok or not to cast \"t\" to int64_t ?\r\n if (lhead->tail.load() != (int64_t)t) continue;\r\n if (h > t) {\r\n int64_t tmp = t;\r\n if (lhead->tail.compare_exchange_strong(tmp, h)) break;\r\n continue;\r\n }\r\n break;\r\n }\r\n }\r\n\r\n int close_crq(Node *rq, const uint64_t tailticket, const int tries) {\r\n if (tries < 10) {\r\n int64_t tmp = tailticket + 1;\r\n return rq->tail.compare_exchange_strong(tmp, (tailticket + 1)|(1ull<<63));\r\n }\r\n else {\r\n return BIT_TEST_AND_SET(&rq->tail, 63);\r\n }\r\n }\r\n\r\n\r\npublic:\r\n LCRQueue(int maxThreads=MAX_THREADS) : maxThreads{maxThreads} {\r\n // Shared object init\r\n Node *sentinel = new Node;\r\n head.store(sentinel, std::memory_order_relaxed);\r\n tail.store(sentinel, std::memory_order_relaxed);\r\n }\r\n\r\n\r\n ~LCRQueue() {\r\n while (dequeue(0) != nullptr); // Drain the queue\r\n delete head.load(); // Delete the last node\r\n }\r\n\r\n static std::string className() { return \"LCRQueue\"; }\r\n\r\n\r\n void enqueue(T* item, const int tid) {\r\n int try_close = 0;\r\n while (true) {\r\n Node* ltail = hp.protectPtr(kHpTail, tail.load(), tid);\r\n if (ltail != tail.load()) continue;\r\n Node *lnext = ltail->next.load();\r\n if (lnext != nullptr) { // Help advance the tail\r\n tail.compare_exchange_strong(ltail, lnext);\r\n continue;\r\n }\r\n\r\n uint64_t tailticket = ltail->tail.fetch_add(1);\r\n if (crq_is_closed(tailticket)) {\r\n Node* newNode = new Node();\r\n // Solo enqueue (superfluous?)\r\n newNode->tail.store(1, std::memory_order_relaxed);\r\n newNode->array[0].val.store(item, std::memory_order_relaxed);\r\n newNode->array[0].idx.store(0, std::memory_order_relaxed);\r\n Node* nullnode = nullptr;\r\n if (ltail->next.compare_exchange_strong(nullnode, newNode)) {// Insert new ring\r\n tail.compare_exchange_strong(ltail, newNode); // Advance the tail\r\n hp.clear(tid);\r\n return;\r\n }\r\n delete newNode;\r\n continue;\r\n }\r\n Cell* cell = <ail->array[tailticket & (RING_SIZE-1)];\r\n uint64_t idx = cell->idx.load();\r\n if (cell->val.load() == nullptr) {\r\n if (node_index(idx) <= tailticket) {\r\n // TODO: is the missing cast before \"t\" ok or not to add?\r\n if ((!node_unsafe(idx) || ltail->head.load() < (int64_t)tailticket)) {\r\n if (CAS2((void**)cell, nullptr, idx, item, tailticket)) {\r\n hp.clear(tid);\r\n return;\r\n }\r\n }\r\n }\r\n }\r\n if (((int64_t)(tailticket - ltail->head.load()) >= (int64_t)RING_SIZE) && close_crq(ltail, tailticket, ++try_close)) continue;\r\n }\r\n }\r\n\r\n\r\n T* dequeue(const int tid) {\r\n while (true) {\r\n Node* lhead = hp.protectPtr(kHpHead, head.load(), tid);\r\n if (lhead != head.load()) continue;\r\n uint64_t headticket = lhead->head.fetch_add(1);\r\n Cell* cell = &lhead->array[headticket & (RING_SIZE-1)];\r\n\r\n int r = 0;\r\n uint64_t tt = 0;\r\n\r\n while (true) {\r\n uint64_t cell_idx = cell->idx.load();\r\n uint64_t unsafe = node_unsafe(cell_idx);\r\n uint64_t idx = node_index(cell_idx);\r\n T* val = cell->val.load();\r\n\r\n if (idx > headticket) break;\r\n\r\n if (val != nullptr) {\r\n if (idx == headticket) {\r\n if (CAS2((void**)cell, val, cell_idx, nullptr, unsafe | (headticket + RING_SIZE))) {\r\n hp.clear(tid);\r\n return val;\r\n }\r\n } else {\r\n if (CAS2((void**)cell, val, cell_idx, val, set_unsafe(idx))) break;\r\n }\r\n } else {\r\n if ((r & ((1ull << 10) - 1)) == 0) tt = lhead->tail.load();\r\n // Optimization: try to bail quickly if queue is closed.\r\n int crq_closed = crq_is_closed(tt);\r\n uint64_t t = tail_index(tt);\r\n if (unsafe) { // Nothing to do, move along\r\n if (CAS2((void**)cell, val, cell_idx, val, unsafe | (headticket + RING_SIZE)))\r\n break;\r\n } else if (t < headticket + 1 || r > 200000 || crq_closed) {\r\n if (CAS2((void**)cell, val, idx, val, headticket + RING_SIZE)) {\r\n if (r > 200000 && tt > RING_SIZE) BIT_TEST_AND_SET(&lhead->tail, 63);\r\n break;\r\n }\r\n } else {\r\n ++r;\r\n }\r\n }\r\n }\r\n\r\n if (tail_index(lhead->tail.load()) <= headticket + 1) {\r\n fixState(lhead);\r\n // try to return empty\r\n Node* lnext = lhead->next.load();\r\n if (lnext == nullptr) {\r\n hp.clear(tid);\r\n return nullptr; // Queue is empty\r\n }\r\n if (tail_index(lhead->tail) <= headticket + 1) {\r\n if (head.compare_exchange_strong(lhead, lnext)) hp.retire(lhead, tid);\r\n }\r\n }\r\n }\r\n }\r\n};\r\n\r\n#endif /* _LCRQ_QUEUE_HP_H_ */\r\n" }, { "path": "datastructures/queues/MichaelScottQueue.hpp", "content": "/******************************************************************************\r\n * Copyright (c) 2014-2016, Pedro Ramalhete, Andreia Correia\r\n * All rights reserved.\r\n *\r\n * Redistribution and use in source and binary forms, with or without\r\n * modification, are permitted provided that the following conditions are met:\r\n * * Redistributions of source code must retain the above copyright\r\n * notice, this list of conditions and the following disclaimer.\r\n * * Redistributions in binary form must reproduce the above copyright\r\n * notice, this list of conditions and the following disclaimer in the\r\n * documentation and/or other materials provided with the distribution.\r\n * * Neither the name of Concurrency Freaks nor the\r\n * names of its contributors may be used to endorse or promote products\r\n * derived from this software without specific prior written permission.\r\n *\r\n * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\r\n * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED\r\n * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\r\n * DISCLAIMED. IN NO EVENT SHALL BE LIABLE FOR ANY\r\n * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\r\n * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\r\n * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND\r\n * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\r\n * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\r\n * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\r\n ******************************************************************************\r\n */\r\n\r\n#ifndef _MICHAEL_SCOTT_QUEUE_HP_H_\r\n#define _MICHAEL_SCOTT_QUEUE_HP_H_\r\n\r\n#include \r\n#include \r\n#include \"HazardPointers.hpp\"\r\n\r\n\r\n/**\r\n *

Michael-Scott Queue

\r\n *\r\n * enqueue algorithm: MS enqueue\r\n * dequeue algorithm: MS dequeue\r\n * Consistency: Linearizable\r\n * enqueue() progress: lock-free\r\n * dequeue() progress: lock-free\r\n * Memory Reclamation: Hazard Pointers (lock-free)\r\n *\r\n *\r\n * Maged Michael and Michael Scott's Queue with Hazard Pointers\r\n *

\r\n * Lock-Free Linked List as described in Maged Michael and Michael Scott's paper:\r\n * {@link http://www.cs.rochester.edu/~scott/papers/1996_PODC_queues.pdf}\r\n * \r\n * Simple, Fast, and Practical Non-Blocking and Blocking Concurrent Queue Algorithms\r\n *

\r\n * The paper on Hazard Pointers is named \"Hazard Pointers: Safe Memory\r\n * Reclamation for Lock-Free objects\" and it is available here:\r\n * http://web.cecs.pdx.edu/~walpole/class/cs510/papers/11.pdf\r\n *\r\n */\r\ntemplate\r\nclass MichaelScottQueue {\r\n\r\nprivate:\r\n struct Node {\r\n T* item;\r\n std::atomic next;\r\n\r\n Node(T* userItem) : item{userItem}, next{nullptr} { }\r\n\r\n bool casNext(Node *cmp, Node *val) {\r\n return next.compare_exchange_strong(cmp, val);\r\n }\r\n };\r\n\r\n bool casTail(Node *cmp, Node *val) {\r\n\t\treturn tail.compare_exchange_strong(cmp, val);\r\n\t}\r\n\r\n bool casHead(Node *cmp, Node *val) {\r\n return head.compare_exchange_strong(cmp, val);\r\n }\r\n\r\n // Pointers to head and tail of the list\r\n alignas(128) std::atomic head;\r\n alignas(128) std::atomic tail;\r\n\r\n static const int MAX_THREADS = 128;\r\n const int maxThreads;\r\n\r\n // We need two hazard pointers for dequeue()\r\n HazardPointers hp {2, maxThreads};\r\n const int kHpTail = 0;\r\n const int kHpHead = 0;\r\n const int kHpNext = 1;\r\n\r\npublic:\r\n MichaelScottQueue(int maxThreads=MAX_THREADS) : maxThreads{maxThreads} {\r\n Node* sentinelNode = new Node(nullptr);\r\n head.store(sentinelNode, std::memory_order_relaxed);\r\n tail.store(sentinelNode, std::memory_order_relaxed);\r\n }\r\n\r\n\r\n ~MichaelScottQueue() {\r\n while (dequeue(0) != nullptr); // Drain the queue\r\n delete head.load(); // Delete the last node\r\n }\r\n\r\n static std::string className() { return \"MichaelScottQueue\"; }\r\n\r\n void enqueue(T* item, const int tid) {\r\n if (item == nullptr) throw std::invalid_argument(\"item can not be nullptr\");\r\n Node* newNode = new Node(item);\r\n while (true) {\r\n Node* ltail = hp.protectPtr(kHpTail, tail, tid);\r\n if (ltail == tail.load()) {\r\n Node* lnext = ltail->next.load();\r\n if (lnext == nullptr) {\r\n // It seems this is the last node, so add the newNode here\r\n // and try to move the tail to the newNode\r\n if (ltail->casNext(nullptr, newNode)) {\r\n casTail(ltail, newNode);\r\n hp.clear(tid);\r\n return;\r\n }\r\n } else {\r\n casTail(ltail, lnext);\r\n }\r\n }\r\n }\r\n }\r\n\r\n\r\n T* dequeue(const int tid) {\r\n Node* node = hp.protect(kHpHead, head, tid);\r\n while (node != tail.load()) {\r\n Node* lnext = hp.protect(kHpNext, node->next, tid);\r\n if (casHead(node, lnext)) {\r\n T* item = lnext->item; // Another thread may clean up lnext after we do hp.clear()\r\n hp.clear(tid);\r\n hp.retire(node, tid);\r\n return item;\r\n }\r\n node = hp.protect(kHpHead, head, tid);\r\n }\r\n hp.clear(tid);\r\n return nullptr; // Queue is empty\r\n }\r\n};\r\n\r\n#endif /* _MICHAEL_SCOTT_QUEUE_HP_H_ */\r\n" }, { "path": "datastructures/queues/OFLFArrayLinkedListQueue.hpp", "content": "/*\r\n * Copyright 2017-2018\r\n * Andreia Correia \r\n * Pedro Ramalhete \r\n * Pascal Felber \r\n * Nachshon Cohen \r\n *\r\n * This work is published under the MIT license. See LICENSE.txt\r\n */\r\n#ifndef _OF_LF_ARRAY_LINKED_LIST_QUEUE_H_\r\n#define _OF_LF_ARRAY_LINKED_LIST_QUEUE_H_\r\n\r\n#include \r\n#include \r\n\r\n#include \"stms/OneFileLF.hpp\"\r\n\r\n/**\r\n *

An Array Linked List Queue using OneFile STM (Lock-Free)

\r\n *\r\n * TODO\r\n *\r\n *\r\n * enqueue algorithm: sequential implementation + MWC\r\n * dequeue algorithm: sequential implementation + MWC\r\n * Consistency: Linearizable\r\n * enqueue() progress: lock-free\r\n * dequeue() progress: lock-free\r\n * Memory Reclamation: Hazard Eras (integrated into MWC)\r\n * enqueue min ops: 2 DCAS + 1 CAS\r\n * dequeue min ops: 1 DCAS + 1 CAS\r\n */\r\ntemplate\r\nclass OFLFArrayLinkedListQueue : public oflf::tmbase {\r\n\r\nprivate:\r\n /*\r\n struct cell {\r\n onefilelf::tmtype val;\r\n } __attribute__ ((aligned (128)));\r\n */\r\n\r\n struct Node : oflf::tmbase {\r\n static const int ITEM_NUM = 1024; // TODO: use a larger ring buffer size here, 1024 for example\r\n oflf::tmtype headidx {0};\r\n //cell items[ITEM_NUM];\r\n oflf::tmtype items[ITEM_NUM];\r\n oflf::tmtype tailidx {0};\r\n oflf::tmtype next {nullptr};\r\n Node(T* item) {\r\n items[0] = item;\r\n tailidx = 1;\r\n headidx = 0;\r\n for (int i = 1; i < ITEM_NUM; i++) items[i] = nullptr;\r\n }\r\n };\r\n\r\n oflf::tmtype head {nullptr};\r\n oflf::tmtype tail {nullptr};\r\n\r\n\r\npublic:\r\n OFLFArrayLinkedListQueue(unsigned int maxThreads=0) {\r\n Node* sentinelNode = new Node(nullptr);\r\n sentinelNode->tailidx = 0;\r\n head = sentinelNode;\r\n tail = sentinelNode;\r\n }\r\n\r\n\r\n ~OFLFArrayLinkedListQueue() {\r\n while (dequeue(0) != nullptr); // Drain the queue\r\n Node* lhead = head;\r\n delete lhead;\r\n }\r\n\r\n\r\n static std::string className() { return \"OF-LF-ArrayLinkedListQueue\"; }\r\n\r\n\r\n /*\r\n * Progress Condition: lock-free\r\n * Always returns true\r\n */\r\n bool enqueue(T* item, const int tid=0) {\r\n if (item == nullptr) throw std::invalid_argument(\"item can not be nullptr\");\r\n return oflf::updateTx([this,item] () -> bool {\r\n Node* ltail = tail;\r\n uint64_t ltailidx = ltail->tailidx;\r\n if (ltailidx < Node::ITEM_NUM) {\r\n ltail->items[ltailidx] = item;\r\n ++ltail->tailidx;\r\n return true;\r\n }\r\n Node* newNode = oflf::tmNew(item);\r\n tail->next = newNode;\r\n tail = newNode;\r\n return true;\r\n });\r\n }\r\n\r\n\r\n /*\r\n * Progress Condition: lock-free\r\n */\r\n T* dequeue(const int tid=0) {\r\n return oflf::updateTx([this] () -> T* {\r\n Node* lhead = head;\r\n uint64_t lheadidx = lhead->headidx;\r\n // Check if queue is empty\r\n if (lhead == tail && lheadidx == tail->tailidx) return nullptr;\r\n if (lheadidx < Node::ITEM_NUM) {\r\n ++lhead->headidx;\r\n return lhead->items[lheadidx];\r\n }\r\n lhead = lhead->next;\r\n oflf::tmDelete(head);\r\n head = lhead;\r\n ++lhead->headidx;\r\n return lhead->items[0];\r\n });\r\n }\r\n};\r\n\r\n#endif /* _OF_LF_ARRAY_LINKED_LIST_QUEUE_H_ */\r\n" }, { "path": "datastructures/queues/OFLFArrayQueue.hpp", "content": "/*\r\n * Copyright 2017-2018\r\n * Andreia Correia \r\n * Pedro Ramalhete \r\n * Pascal Felber \r\n * Nachshon Cohen \r\n *\r\n * This work is published under the MIT license. See LICENSE.txt\r\n */\r\n#ifndef _OFLF_STM_ARRAY_QUEUE_H_\r\n#define _OFLF_STM_ARRAY_QUEUE_H_\r\n\r\n#include \r\n#include \r\n\r\n#include \"stms/OneFileLF.hpp\"\r\n\r\n/**\r\n *

An Array Queue

\r\n *\r\n */\r\ntemplate\r\nclass OFLFArrayQueue : public oflf::tmbase {\r\n\r\nprivate:\r\n static const int MAX_ITEMS = 2048;\r\n oflf::tmtype headidx {0};\r\n oflf::tmtype items[MAX_ITEMS];\r\n oflf::tmtype tailidx {0};\r\n\r\n\r\npublic:\r\n OFLFArrayQueue(unsigned int maxThreads=0) {\r\n oflf::updateTx([this] () {\r\n for (int i = 0; i < MAX_ITEMS; i++) items[i] = nullptr;\r\n return true;\r\n });\r\n }\r\n\r\n\r\n ~OFLFArrayQueue() { }\r\n\r\n\r\n static std::string className() { return \"OF-LF-ArrayQueue\"; }\r\n\r\n\r\n /*\r\n * Progress Condition: blocking\r\n * Always returns true\r\n */\r\n bool enqueue(T* item, const int tid=0) {\r\n if (item == nullptr) throw std::invalid_argument(\"item can not be nullptr\");\r\n return oflf::updateTx([this,item] () -> bool {\r\n if (tailidx >= headidx+MAX_ITEMS) return false; // queue is full\r\n items[tailidx % MAX_ITEMS] = item;\r\n ++tailidx;\r\n return true;\r\n });\r\n }\r\n\r\n\r\n /*\r\n * Progress Condition: blocking\r\n */\r\n T* dequeue(const int tid=0) {\r\n return oflf::updateTx([this] () -> T* {\r\n if (tailidx == headidx) return nullptr; // queue is empty\r\n T* item = items[headidx % MAX_ITEMS];\r\n ++headidx;\r\n return item;\r\n });\r\n }\r\n};\r\n\r\n#endif /* _OF_LF_STM_ARRAY_QUEUE_H_ */\r\n" }, { "path": "datastructures/queues/OFLFLinkedListQueue.hpp", "content": "/*\r\n * Copyright 2017-2018\r\n * Andreia Correia \r\n * Pedro Ramalhete \r\n * Pascal Felber \r\n * Nachshon Cohen \r\n *\r\n * This work is published under the MIT license. See LICENSE.txt\r\n */\r\n#ifndef _OF_LF_LINKED_LIST_QUEUE_H_\r\n#define _OF_LF_LINKED_LIST_QUEUE_H_\r\n\r\n#include \r\n#include \r\n\r\n#include \"stms/OneFileLF.hpp\"\r\n\r\n/**\r\n *

A Linked List queue using OneFile STM (Lock-Free)

\r\n *\r\n * enqueue algorithm: sequential implementation + OFLF\r\n * dequeue algorithm: sequential implementation + OFLF\r\n * Consistency: Linearizable\r\n * enqueue() progress: lock-free\r\n * dequeue() progress: lock-free\r\n * Memory Reclamation: lock-free Hazard Eras (integrated into OFLF)\r\n * enqueue min ops: 2 DCAS + 1 CAS\r\n * dequeue min ops: 1 DCAS + 1 CAS\r\n */\r\ntemplate\r\nclass OFLFLinkedListQueue : public oflf::tmbase {\r\n\r\nprivate:\r\n struct Node : oflf::tmbase {\r\n T* item;\r\n oflf::tmtype next {nullptr};\r\n Node(T* userItem) : item{userItem} { }\r\n };\r\n\r\n oflf::tmtype head {nullptr};\r\n oflf::tmtype tail {nullptr};\r\n\r\n\r\npublic:\r\n OFLFLinkedListQueue(unsigned int maxThreads=0) {\r\n Node* sentinelNode = oflf::tmNew(nullptr);\r\n head = sentinelNode;\r\n tail = sentinelNode;\r\n }\r\n\r\n\r\n ~OFLFLinkedListQueue() {\r\n while (dequeue() != nullptr); // Drain the queue\r\n Node* lhead = head;\r\n oflf::tmDelete(lhead);\r\n }\r\n\r\n\r\n static std::string className() { return \"OF-LF-LinkedListQueue\"; }\r\n\r\n\r\n /*\r\n * Progress Condition: lock-free\r\n * Always returns true\r\n */\r\n bool enqueue(T* item, const int tid=0) {\r\n if (item == nullptr) throw std::invalid_argument(\"item can not be nullptr\");\r\n Node* newNode = oflf::tmNew(item); // Let's allocate outside the transaction, less overhead\r\n return oflf::updateTx([this,newNode] () -> bool {\r\n tail->next = newNode;\r\n tail = newNode;\r\n return true;\r\n });\r\n }\r\n\r\n\r\n /*\r\n * Progress Condition: lock-free\r\n */\r\n T* dequeue(const int tid=0) {\r\n return oflf::updateTx([this] () -> T* {\r\n Node* lhead = head;\r\n if (lhead == tail) return nullptr;\r\n head = lhead->next;\r\n oflf::tmDelete(lhead);\r\n return head->item;\r\n });\r\n }\r\n};\r\n\r\n#endif /* _OF_LF_LINKED_LIST_QUEUE_H_ */\r\n" }, { "path": "datastructures/queues/OFWFArrayLinkedListQueue.hpp", "content": "/*\r\n * Copyright 2017-2018\r\n * Andreia Correia \r\n * Pedro Ramalhete \r\n * Pascal Felber \r\n * Nachshon Cohen \r\n *\r\n * This work is published under the MIT license. See LICENSE.txt\r\n */\r\n#ifndef _OF_WF_ARRAY_LINKED_LIST_QUEUE_H_\r\n#define _OF_WF_ARRAY_LINKED_LIST_QUEUE_H_\r\n\r\n#include \r\n#include \r\n\r\n#include \"stms/OneFileWF.hpp\"\r\n\r\n/**\r\n *

An Array Linked List Queue using OneFile STM (Wait-Free)

\r\n *\r\n * TODO\r\n *\r\n *\r\n * enqueue algorithm: sequential implementation + MWC\r\n * dequeue algorithm: sequential implementation + MWC\r\n * Consistency: Linearizable\r\n * enqueue() progress: lock-free\r\n * dequeue() progress: lock-free\r\n * Memory Reclamation: Hazard Eras (integrated into MWC)\r\n * enqueue min ops: 2 DCAS + 1 CAS\r\n * dequeue min ops: 1 DCAS + 1 CAS\r\n */\r\ntemplate\r\nclass OFWFArrayLinkedListQueue : public ofwf::tmbase {\r\n\r\nprivate:\r\n struct Node : ofwf::tmbase {\r\n static const int ITEM_NUM = 1024;\r\n ofwf::tmtype headidx {0};\r\n ofwf::tmtype items[ITEM_NUM];\r\n ofwf::tmtype tailidx {0};\r\n ofwf::tmtype next {nullptr};\r\n Node(T* item) {\r\n items[0] = item;\r\n tailidx = 1;\r\n headidx = 0;\r\n for (int i = 1; i < ITEM_NUM; i++) items[i] = nullptr;\r\n }\r\n };\r\n\r\n ofwf::tmtype head {nullptr};\r\n ofwf::tmtype tail {nullptr};\r\n\r\n\r\npublic:\r\n OFWFArrayLinkedListQueue(unsigned int maxThreads=0) {\r\n Node* sentinelNode = new Node(nullptr);\r\n sentinelNode->tailidx = 0;\r\n head = sentinelNode;\r\n tail = sentinelNode;\r\n }\r\n\r\n\r\n ~OFWFArrayLinkedListQueue() {\r\n while (dequeue(0) != nullptr); // Drain the queue\r\n Node* lhead = head;\r\n delete lhead;\r\n }\r\n\r\n\r\n static std::string className() { return \"OF-WF-ArrayLinkedListQueue\"; }\r\n\r\n\r\n /*\r\n * Progress Condition: lock-free\r\n * Always returns true\r\n */\r\n bool enqueue(T* item, const int tid=0) {\r\n if (item == nullptr) throw std::invalid_argument(\"item can not be nullptr\");\r\n return ofwf::updateTx([this,item] () -> bool {\r\n Node* ltail = tail;\r\n uint64_t ltailidx = ltail->tailidx;\r\n if (ltailidx < Node::ITEM_NUM) {\r\n ltail->items[ltailidx] = item;\r\n ++ltail->tailidx;\r\n return true;\r\n }\r\n Node* newNode = ofwf::tmNew(item);\r\n tail->next = newNode;\r\n tail = newNode;\r\n return true;\r\n });\r\n }\r\n\r\n\r\n /*\r\n * Progress Condition: lock-free\r\n */\r\n T* dequeue(const int tid=0) {\r\n return ofwf::updateTx([this] () -> T* {\r\n Node* lhead = head;\r\n uint64_t lheadidx = lhead->headidx;\r\n // Check if queue is empty\r\n if (lhead == tail && lheadidx == tail->tailidx) return nullptr;\r\n if (lheadidx < Node::ITEM_NUM) {\r\n ++lhead->headidx;\r\n return lhead->items[lheadidx];\r\n }\r\n lhead = lhead->next;\r\n ofwf::tmDelete(head);\r\n head = lhead;\r\n ++lhead->headidx;\r\n return lhead->items[0];\r\n });\r\n }\r\n};\r\n\r\n#endif /* _OF_WF_ARRAY_LINKED_LIST_QUEUE_H_ */\r\n" }, { "path": "datastructures/queues/OFWFLinkedListQueue.hpp", "content": "/*\r\n * Copyright 2017-2018\r\n * Andreia Correia \r\n * Pedro Ramalhete \r\n * Pascal Felber \r\n * Nachshon Cohen \r\n *\r\n * This work is published under the MIT license. See LICENSE.txt\r\n */\r\n#ifndef _OF_WF_LINKED_LIST_QUEUE_H_\r\n#define _OF_WF_LINKED_LIST_QUEUE_H_\r\n\r\n#include \r\n#include \r\n\r\n#include \"stms/OneFileWF.hpp\"\r\n\r\n/**\r\n *

A Linked List queue using OneFile STM (Wait-Free)

\r\n *\r\n * enqueue algorithm: sequential implementation + OFWF\r\n * dequeue algorithm: sequential implementation + OFWF\r\n * Consistency: Linearizable\r\n * enqueue() progress: wait-free\r\n * dequeue() progress: wait-free\r\n * Memory Reclamation: wait-free Hazard Eras (integrated into OFWF)\r\n * enqueue min ops: 3 DCAS + 1 CAS\r\n * dequeue min ops: 2 DCAS + 1 CAS\r\n */\r\ntemplate\r\nclass OFWFLinkedListQueue : public ofwf::tmbase {\r\n\r\nprivate:\r\n struct Node : ofwf::tmbase {\r\n T* item;\r\n ofwf::tmtype next;\r\n Node(T* userItem) : item{userItem}, next{nullptr} { }\r\n };\r\n\r\n ofwf::tmtype head {nullptr};\r\n ofwf::tmtype tail {nullptr};\r\n\r\n\r\npublic:\r\n OFWFLinkedListQueue(unsigned int maxThreads=0) {\r\n Node* sentinelNode = ofwf::tmNew(nullptr);\r\n head = sentinelNode;\r\n tail = sentinelNode;\r\n }\r\n\r\n\r\n ~OFWFLinkedListQueue() {\r\n while (dequeue() != nullptr); // Drain the queue\r\n Node* lhead = head;\r\n ofwf::tmDelete(lhead);\r\n }\r\n\r\n\r\n static std::string className() { return \"OF-WF-LinkedListQueue\"; }\r\n\r\n\r\n /*\r\n * Progress Condition: wait-free bounded\r\n * Always returns true\r\n */\r\n bool enqueue(T* item, const int tid=0) {\r\n if (item == nullptr) throw std::invalid_argument(\"item can not be nullptr\");\r\n Node* newNode = ofwf::tmNew(item); // Let's allocate outside the transaction, less overhead\r\n return ofwf::updateTx([this,newNode] () -> bool {\r\n tail->next = newNode;\r\n tail = newNode;\r\n return true;\r\n });\r\n }\r\n\r\n\r\n /*\r\n * Progress Condition: wait-free bounded\r\n */\r\n T* dequeue(const int tid=0) {\r\n return (T*)ofwf::updateTx([this] () -> T* {\r\n Node* lhead = head;\r\n if (lhead == tail) return nullptr;\r\n head = lhead->next;\r\n ofwf::tmDelete(lhead);\r\n return head->item;\r\n });\r\n }\r\n};\r\n\r\n#endif /* _OF_WF_LINKED_LIST_QUEUE_H_ */\r\n" }, { "path": "datastructures/queues/README.md", "content": "# Queues #\r\n\r\nThis folder contains multiple multi-producer-multi-consumer queue implementations, all of them with integrated memory reclamation having the same progress condition:\r\n\r\n- FAAArrayQueue: Memory unbounded, lock-free, one array per node, hazard pointers\r\nhttp://...\r\n\r\n- LCRQueue: Memory unbounded, lock-free, one array per node, hazard pointers, can re-use entries in some situations\r\nhttp://\r\n\r\n- OFLFLinkedListqueue: Memory unbounded, lock-free, one entry per node, hazard eras\r\nUses OneFile STM (Lock-Free)\r\n\r\n- OFWFLinkedListqueue: Memory unbounded, wait-free bounded, one entry per node, hazard eras\r\nUses OneFile STM (Wait-Free)\r\n\r\n- SimQueue: Memory unbounded, wait-free bounded, one entry per node, modified hazard pointers\r\nhttp://\r\n\r\n- TurnQueue: Memory unbounded, wait-free bounded, one entry per node, hazard pointers\r\nhttp:// \r\n" }, { "path": "datastructures/queues/SimQueue.hpp", "content": "/******************************************************************************\r\n * Copyright (c) 2014-2016, Pedro Ramalhete, Andreia Correia\r\n * All rights reserved.\r\n *\r\n * Redistribution and use in source and binary forms, with or without\r\n * modification, are permitted provided that the following conditions are met:\r\n * * Redistributions of source code must retain the above copyright\r\n * notice, this list of conditions and the following disclaimer.\r\n * * Redistributions in binary form must reproduce the above copyright\r\n * notice, this list of conditions and the following disclaimer in the\r\n * documentation and/or other materials provided with the distribution.\r\n * * Neither the name of Concurrency Freaks nor the\r\n * names of its contributors may be used to endorse or promote products\r\n * derived from this software without specific prior written permission.\r\n *\r\n * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\r\n * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED\r\n * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\r\n * DISCLAIMED. IN NO EVENT SHALL BE LIABLE FOR ANY\r\n * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\r\n * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\r\n * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND\r\n * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\r\n * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\r\n * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\r\n ******************************************************************************\r\n */\r\n\r\n#ifndef _CR_SIM_QUEUE_HP_H_\r\n#define _CR_SIM_QUEUE_HP_H_\r\n\r\n#include \r\n#include \r\n#include \"HazardPointersSimQueue.hpp\"\r\n\r\n\r\n/**\r\n *

Sim Queue

\r\n *\r\n * Based on the SimQueue (FK queue)\r\n *\r\n * http://thalis.cs.uoi.gr/tech_reports/publications/TR2011-01.pdf\r\n *\r\n * enqueue algorithm: P-Sim\r\n * dequeue algorithm: P-Sim\r\n * Consistency: Linearizable\r\n * enqueue() progress: wait-free bounded O(N_threads)\r\n * dequeue() progress: wait-free bounded O(N_threads)\r\n * Memory Reclamation: Hazard Pointers with custom scanner for Nodes. EnqState and DeqState re-usage.\r\n *\r\n *

\r\n * The paper on Hazard Pointers is named \"Hazard Pointers: Safe Memory\r\n * Reclamation for Lock-Free objects\" and it is available here:\r\n * http://web.cecs.pdx.edu/~walpole/class/cs510/papers/11.pdf\r\n *\r\n */\r\ntemplate\r\nclass SimQueue {\r\n\r\nprivate:\r\n static const int MAX_THREADS = 128;\r\n\r\n struct Node {\r\n T* item;\r\n std::atomic next {nullptr};\r\n Node(T* item) : item{item} { }\r\n };\r\n\r\n\r\n struct EnqState {\r\n std::atomic tail {nullptr}; // link_a\r\n std::atomic nextNode {nullptr}; // link_b\r\n std::atomic nextTail {nullptr}; // ptr\r\n std::atomic applied[MAX_THREADS];\r\n\r\n EnqState() {\r\n for(int i=0; i < MAX_THREADS; i++){\r\n applied[i].store(false, std::memory_order_relaxed);\r\n }\r\n }\r\n };\r\n\r\n\r\n struct DeqState {\r\n std::atomic head {nullptr};\r\n std::atomic items[MAX_THREADS];\r\n std::atomic applied[MAX_THREADS];\r\n\r\n DeqState() {\r\n for(int i=0; i < MAX_THREADS; i++){\r\n applied[i].store(false, std::memory_order_relaxed);\r\n items[i].store(nullptr, std::memory_order_relaxed);\r\n }\r\n }\r\n };\r\n\r\n\r\n typedef union pointer_t {\r\n struct StructData{\r\n int64_t seq : 48;\r\n int64_t index: 16;\r\n } u; // struct_data\r\n int64_t raw; // raw_data\r\n } pointer_t;\r\n\r\n\r\n const int maxThreads;\r\n\r\n alignas(128) std::atomic enqPointer;\r\n alignas(128) std::atomic deqPointer;\r\n // Enqueue requests\r\n alignas(128) std::atomic items[MAX_THREADS]; // Always access relaxed\r\n alignas(128) std::atomic enqueuers[MAX_THREADS];\r\n // Re-usable EnqState instances\r\n alignas(128) EnqState enqReused[MAX_THREADS*2];\r\n // Dequeue requests\r\n alignas(128) std::atomic dequeuers[MAX_THREADS];\r\n // Re-usable DeqState instances\r\n alignas(128) DeqState deqReused[MAX_THREADS*2];\r\n alignas(128) Node* pool[MAX_THREADS][MAX_THREADS];\r\n\r\n\r\n // Passed to Hazard Pointers\r\n std::function find = [this](Node* ptr) {\r\n pointer_t lpointer = enqPointer.load();\r\n if (enqReused[lpointer.u.index].tail.load() == ptr) return true;\r\n /*\r\n lpointer = deqPointer.load();\r\n if (deqReused[lpointer.u.index].head.load() == ptr) return true;\r\n */\r\n return false;\r\n };\r\n\r\n HazardPointersSimQueue hp {find, 1, maxThreads};\r\n const int kHpTail = 0;\r\n const int kHpNode = 0;\r\n\r\n Node* sentinel = new Node(nullptr);\r\n\r\npublic:\r\n SimQueue(int maxThreads=MAX_THREADS) : maxThreads(maxThreads) {\r\n for (int i = 0; i < maxThreads; i++) {\r\n enqueuers[i].store(false, std::memory_order_relaxed);\r\n dequeuers[i].store(false, std::memory_order_relaxed);\r\n for(int j=0;jtail.load(), tid);\r\n Node* lnext = lstate->nextNode.load(); // No need for HP because we don't dereference it\r\n Node* lnextTail = lstate->nextTail.load(); // No need for HP\r\n if (lpointer.raw != enqPointer.load().raw) continue;\r\n\r\n // Advance the tail if needed\r\n if (ltail->next.load() != lnext) {\r\n ltail->next.store(lnext, std::memory_order_release);\r\n }\r\n // Check if my request has been done\r\n if (lstate->applied[tid].load() == newrequest) {\r\n if (lpointer.raw == enqPointer.load().raw) break;\r\n }\r\n // Help opened enqueue requests, starting from zero\r\n Node* first = nullptr;\r\n Node* node = nullptr;\r\n const int myIndex = (lpointer.u.index == 2*tid) ? 2*tid+1 : 2*tid ;\r\n EnqState* const myState = &enqReused[myIndex];\r\n int numNodes = 0;\r\n for (int j = 0; j < maxThreads; j++) {\r\n // Check if it is an open request\r\n const bool enqj = enqueuers[j].load();\r\n myState->applied[j].store(enqj, std::memory_order_relaxed);\r\n if (enqj == lstate->applied[j].load()) continue;\r\n Node* prev = node;\r\n node = pool[tid][numNodes++];\r\n node->item = items[j].load(std::memory_order_relaxed);\r\n if (first == nullptr) {\r\n first = node;\r\n } else {\r\n prev->next.store(node, std::memory_order_relaxed);\r\n }\r\n if (lpointer.raw != enqPointer.load().raw) break;\r\n }\r\n\r\n // Try to apply the new sublist\r\n if (lpointer.raw != enqPointer.load().raw) continue;\r\n node->next.store(nullptr, std::memory_order_relaxed);\r\n myState->tail.store(lnextTail, std::memory_order_relaxed);\r\n myState->nextNode.store(first, std::memory_order_relaxed);\r\n myState->nextTail.store(node, std::memory_order_relaxed);\r\n pointer_t myPointer;\r\n myPointer.u.seq = lpointer.u.seq + 1;\r\n myPointer.u.index = myIndex;\r\n if (enqPointer.compare_exchange_strong(lpointer, myPointer)) {\r\n for (int k = 0; k < numNodes; k++) { // Refill pool\r\n pool[tid][k] = new Node(nullptr);\r\n }\r\n }\r\n }\r\n hp.clear(tid);\r\n }\r\n\r\n\r\n\r\n /**\r\n * Progress condition: wait-free bounded\r\n *\r\n * We use just one HP index, but it was though to get there.\r\n */\r\n T* dequeue(const int tid) {\r\n // Publish dequeue request\r\n bool newrequest = !dequeuers[tid].load(std::memory_order_relaxed);\r\n dequeuers[tid].store(newrequest);\r\n for (int iter = 0; iter < 2; iter++) {\r\n pointer_t lpointer = deqPointer.load();\r\n DeqState* lstate = &deqReused[lpointer.u.index];\r\n // Check if my request has been done\r\n if (lstate->applied[tid].load() == newrequest) {\r\n if (lpointer.raw == deqPointer.load().raw) break;\r\n }\r\n // Help opened dequeue requests, starting from turn+1\r\n Node* newHead = hp.protectPtr(kHpNode, lstate->head, tid);\r\n if (lpointer.raw != deqPointer.load().raw) continue;\r\n const int myIndex = (lpointer.u.index == 2*tid) ? 2*tid+1 : 2*tid ;\r\n DeqState* const myState = &deqReused[myIndex];\r\n Node* node = newHead;\r\n for (int j = 0; j < maxThreads; j++) {\r\n // Check if it is an open request\r\n const bool applied = lstate->applied[j].load();\r\n if (dequeuers[j].load() == applied) {\r\n myState->items[j].store(lstate->items[j], std::memory_order_relaxed);\r\n myState->applied[j].store(applied, std::memory_order_relaxed);\r\n continue;\r\n }\r\n myState->applied[j].store(!applied, std::memory_order_relaxed);\r\n if (node->next.load() == nullptr) {\r\n myState->items[j].store(nullptr,std::memory_order_relaxed);\r\n } else {\r\n node = hp.protectPtr(kHpNode, node->next, tid);\r\n if (lpointer.raw != deqPointer.load().raw) break;\r\n myState->items[j].store(node->item, std::memory_order_relaxed);\r\n newHead = node;\r\n }\r\n }\r\n if (lpointer.raw != deqPointer.load().raw) continue;\r\n pointer_t newDeqIndex;\r\n newDeqIndex.u.seq = lpointer.u.seq + 1;\r\n newDeqIndex.u.index = myIndex;\r\n myState->head.store(newHead, std::memory_order_relaxed);\r\n node = lstate->head;\r\n if (deqPointer.compare_exchange_strong(lpointer, newDeqIndex)) {\r\n while (node != newHead) {\r\n Node* next = node->next.load();\r\n hp.retire(node,tid);\r\n node = next;\r\n }\r\n break;\r\n }\r\n }\r\n hp.clear(tid);\r\n return deqReused[deqPointer.load().u.index].items[tid].load();\r\n }\r\n};\r\n\r\n#endif /* _SIM_QUEUE_HP_H_ */\r\n" }, { "path": "datastructures/queues/TinySTMArrayLinkedListQueue.hpp", "content": "/******************************************************************************\r\n * Copyright (c) 2014-2017, Pedro Ramalhete, Andreia Correia\r\n * All rights reserved.\r\n *\r\n * Redistribution and use in source and binary forms, with or without\r\n * modification, are permitted provided that the following conditions are met:\r\n * * Redistributions of source code must retain the above copyright\r\n * notice, this list of conditions and the following disclaimer.\r\n * * Redistributions in binary form must reproduce the above copyright\r\n * notice, this list of conditions and the following disclaimer in the\r\n * documentation and/or other materials provided with the distribution.\r\n * * Neither the name of Concurrency Freaks nor the\r\n * names of its contributors may be used to endorse or promote products\r\n * derived from this software without specific prior written permission.\r\n *\r\n * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\r\n * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED\r\n * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\r\n * DISCLAIMED. IN NO EVENT SHALL BE LIABLE FOR ANY\r\n * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\r\n * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\r\n * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND\r\n * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\r\n * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\r\n * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\r\n ******************************************************************************\r\n */\r\n\r\n#ifndef _TINY_STM_ARRAY_LINKED_LIST_QUEUE_H_\r\n#define _TINY_STM_ARRAY_LINKED_LIST_QUEUE_H_\r\n\r\n#include \r\n#include \r\n\r\n#include \"stms/TinySTM.hpp\"\r\n\r\n/**\r\n *

An Array Linked List Queue using Tiny STM

\r\n */\r\ntemplate\r\nclass TinySTMArrayLinkedListQueue {\r\n\r\nprivate:\r\n struct Node : tinystm::tmbase {\r\n static const int ITEM_NUM = 1024;\r\n tinystm::tmtype headidx {0};\r\n tinystm::tmtype items[ITEM_NUM];\r\n tinystm::tmtype tailidx {0};\r\n tinystm::tmtype next {nullptr};\r\n Node(T* item) {\r\n items[0] = item;\r\n tailidx = 1;\r\n headidx = 0;\r\n for (int i = 1; i < ITEM_NUM; i++) items[i] = nullptr;\r\n }\r\n };\r\n\r\n tinystm::tmtype head {nullptr};\r\n tinystm::tmtype tail {nullptr};\r\n\r\n\r\npublic:\r\n TinySTMArrayLinkedListQueue(unsigned int maxThreads=0) {\r\n tinystm::updateTx([this] () {\r\n Node* sentinelNode = tinystm::tmNew(nullptr);\r\n sentinelNode->tailidx = 0;\r\n head = sentinelNode;\r\n tail = sentinelNode;\r\n return true;\r\n });\r\n }\r\n\r\n\r\n ~TinySTMArrayLinkedListQueue() {\r\n while (dequeue() != nullptr); // Drain the queue\r\n tinystm::updateTx([this] () {\r\n Node* lhead = head;\r\n tinystm::tmDelete(lhead);\r\n return true;\r\n });\r\n }\r\n\r\n\r\n static std::string className() { return \"TinySTM-ArrayLinkedListQueue\"; }\r\n\r\n\r\n /*\r\n * Progress Condition: blocking\r\n * Always returns true\r\n */\r\n bool enqueue(T* item, const int tid=0) {\r\n if (item == nullptr) throw std::invalid_argument(\"item can not be nullptr\");\r\n return tinystm::updateTx([this,item] () -> bool {\r\n Node* ltail = tail;\r\n uint64_t ltailidx = ltail->tailidx;\r\n if (ltailidx < Node::ITEM_NUM) {\r\n ltail->items[ltailidx] = item;\r\n ++ltail->tailidx;\r\n return true;\r\n }\r\n Node* newNode = tinystm::tmNew(item);\r\n tail->next = newNode;\r\n tail = newNode;\r\n return true;\r\n });\r\n }\r\n\r\n\r\n /*\r\n * Progress Condition: blocking\r\n */\r\n T* dequeue(const int tid=0) {\r\n return tinystm::updateTx([this] () -> T* {\r\n Node* lhead = head;\r\n uint64_t lheadidx = lhead->headidx;\r\n // Check if queue is empty\r\n if (lhead == tail && lheadidx == tail->tailidx) return nullptr;\r\n if (lheadidx < Node::ITEM_NUM) {\r\n ++lhead->headidx;\r\n return lhead->items[lheadidx];\r\n }\r\n lhead = lhead->next;\r\n tinystm::tmDelete(head.load());\r\n head = lhead;\r\n ++lhead->headidx;\r\n return lhead->items[0];\r\n });\r\n }\r\n};\r\n\r\n#endif /* _TINY_STM_ARRAY_LINKED_LIST_QUEUE_H_ */\r\n" }, { "path": "datastructures/queues/TinySTMLinkedListQueue.hpp", "content": "/******************************************************************************\r\n * Copyright (c) 2014-2018, Pedro Ramalhete, Andreia Correia\r\n * All rights reserved.\r\n *\r\n * Redistribution and use in source and binary forms, with or without\r\n * modification, are permitted provided that the following conditions are met:\r\n * * Redistributions of source code must retain the above copyright\r\n * notice, this list of conditions and the following disclaimer.\r\n * * Redistributions in binary form must reproduce the above copyright\r\n * notice, this list of conditions and the following disclaimer in the\r\n * documentation and/or other materials provided with the distribution.\r\n * * Neither the name of Concurrency Freaks nor the\r\n * names of its contributors may be used to endorse or promote products\r\n * derived from this software without specific prior written permission.\r\n *\r\n * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\r\n * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED\r\n * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\r\n * DISCLAIMED. IN NO EVENT SHALL BE LIABLE FOR ANY\r\n * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\r\n * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\r\n * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND\r\n * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\r\n * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\r\n * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\r\n ******************************************************************************\r\n */\r\n\r\n#ifndef _TINY_STM_LINKED_LIST_QUEUE_H_\r\n#define _TINY_STM_LINKED_LIST_QUEUE_H_\r\n\r\n#include \r\n#include \r\n\r\n#include \"stms/TinySTM.hpp\"\r\n\r\n/**\r\n *

A Linked List queue using Tiny STM

\r\n\r\n */\r\ntemplate\r\nclass TinySTMLinkedListQueue : public tinystm::tmbase {\r\n\r\nprivate:\r\n struct Node : tinystm::tmbase {\r\n T* item;\r\n tinystm::tmtype next {nullptr};\r\n Node(T* userItem) : item{userItem} { }\r\n };\r\n\r\n tinystm::tmtype head {nullptr};\r\n tinystm::tmtype tail {nullptr};\r\n\r\n\r\npublic:\r\n TinySTMLinkedListQueue(unsigned int maxThreads=0) {\r\n tinystm::updateTx([this] () {\r\n Node* sentinelNode = tinystm::tmNew(nullptr);\r\n head = sentinelNode;\r\n tail = sentinelNode;\r\n return true;\r\n });\r\n }\r\n\r\n\r\n ~TinySTMLinkedListQueue() {\r\n while (dequeue() != nullptr); // Drain the queue\r\n tinystm::updateTx([this] () {\r\n Node* lhead = head;\r\n tinystm::tmDelete(lhead);\r\n return true;\r\n });\r\n }\r\n\r\n\r\n static std::string className() { return \"TinySTM-LinkedListQueue\"; }\r\n\r\n\r\n /*\r\n * Progress Condition: blocking\r\n * Always returns true\r\n */\r\n bool enqueue(T* item, const int tid=0) {\r\n if (item == nullptr) throw std::invalid_argument(\"item can not be nullptr\");\r\n return tinystm::updateTx([this,item] () -> bool {\r\n Node* newNode = tinystm::tmNew(item);\r\n tail->next = newNode;\r\n tail = newNode;\r\n return true;\r\n });\r\n }\r\n\r\n\r\n /*\r\n * Progress Condition: blocking\r\n */\r\n T* dequeue(const int tid=0) {\r\n return tinystm::updateTx([this] () -> T* {\r\n Node* lhead = head;\r\n if (lhead == tail) return nullptr;\r\n head = lhead->next;\r\n tinystm::tmDelete(lhead);\r\n return head->item;\r\n });\r\n }\r\n};\r\n\r\n#endif /* _TINY_STM_LINKED_LIST_QUEUE_H_ */\r\n" }, { "path": "datastructures/queues/TurnQueue.hpp", "content": "/******************************************************************************\r\n * Copyright (c) 2014-2016, Pedro Ramalhete, Andreia Correia\r\n * All rights reserved.\r\n *\r\n * Redistribution and use in source and binary forms, with or without\r\n * modification, are permitted provided that the following conditions are met:\r\n * * Redistributions of source code must retain the above copyright\r\n * notice, this list of conditions and the following disclaimer.\r\n * * Redistributions in binary form must reproduce the above copyright\r\n * notice, this list of conditions and the following disclaimer in the\r\n * documentation and/or other materials provided with the distribution.\r\n * * Neither the name of Concurrency Freaks nor the\r\n * names of its contributors may be used to endorse or promote products\r\n * derived from this software without specific prior written permission.\r\n *\r\n * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\r\n * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED\r\n * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\r\n * DISCLAIMED. IN NO EVENT SHALL BE LIABLE FOR ANY\r\n * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\r\n * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\r\n * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND\r\n * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\r\n * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\r\n * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\r\n ******************************************************************************\r\n */\r\n\r\n#ifndef _TURN_QUEUE_HP_H_\r\n#define _TURN_QUEUE_HP_H_\r\n\r\n#include \r\n#include \r\n#include \"HazardPointers.hpp\"\r\n\r\n\r\n/**\r\n *

Turn Queue

\r\n *\r\n * A concurrent wait-free queue that is Multi-Producer-Multi-Consumer and does\r\n * its own wait-free memory reclamation.\r\n * Based on the paper \"A Wait-Free Queue with Wait-Free Memory Reclamation\"\r\n * https://github.com/pramalhe/ConcurrencyFreaks/tree/master/papers/crturnqueue-2016.pdf\r\n *\r\n *

\r\n * Enqueue algorithm: CR Turn enqueue\r\n * Dequeue algorithm: CR Turn dequeue\r\n * Consistency: Linearizable\r\n * enqueue() progress: wait-free bounded O(N_threads)\r\n * dequeue() progress: wait-free bounded O(N_threads)\r\n * Memory Reclamation: Hazard Pointers (wait-free)\r\n *\r\n *

\r\n * The paper on Hazard Pointers is named \"Hazard Pointers: Safe Memory\r\n * Reclamation for Lock-Free objects\" and it is available here:\r\n * http://web.cecs.pdx.edu/~walpole/class/cs510/papers/11.pdf\r\n *\r\n * @author Andreia Correia\r\n * @author Pedro Ramalhete\r\n */\r\ntemplate\r\nclass TurnQueue {\r\n\r\nprivate:\r\n struct Node {\r\n T* item;\r\n const int enqTid;\r\n std::atomic deqTid;\r\n std::atomic next;\r\n\r\n Node(T* item, int tid) : item{item}, enqTid{tid}, deqTid{IDX_NONE}, next{nullptr} { }\r\n\r\n bool casDeqTid(int cmp, int val) {\r\n \t return deqTid.compare_exchange_strong(cmp, val);\r\n }\r\n };\r\n\r\n static const int IDX_NONE = -1;\r\n static const int MAX_THREADS = 128;\r\n const int maxThreads;\r\n\r\n // Pointers to head and tail of the list\r\n alignas(128) std::atomic head;\r\n alignas(128) std::atomic tail;\r\n // Enqueue requests\r\n alignas(128) std::atomic enqueuers[MAX_THREADS];\r\n // Dequeue requests\r\n alignas(128) std::atomic deqself[MAX_THREADS];\r\n alignas(128) std::atomic deqhelp[MAX_THREADS];\r\n\r\n\r\n HazardPointers hp {3, maxThreads}; // We need three hazard pointers\r\n const int kHpTail = 0;\r\n const int kHpHead = 0;\r\n const int kHpNext = 1;\r\n const int kHpDeq = 2;\r\n\r\n Node* sentinelNode = new Node(nullptr, 0);\r\n\r\n\r\n /**\r\n * Called only from dequeue()\r\n *\r\n * Search for the next request to dequeue and assign it to lnext.deqTid\r\n * It is only a request to dequeue if deqself[i] equals deqhelp[i].\r\n */\r\n inline int searchNext(Node* lhead, Node* lnext) {\r\n const int turn = lhead->deqTid.load();\r\n for (int idx=turn+1; idx < turn+maxThreads+1; idx++) {\r\n const int idDeq = idx%maxThreads;\r\n if (deqself[idDeq].load() != deqhelp[idDeq].load()) continue;\r\n if (lnext->deqTid.load() == IDX_NONE) lnext->casDeqTid(IDX_NONE, idDeq);\r\n break;\r\n }\r\n return lnext->deqTid.load();\r\n }\r\n\r\n\r\n /**\r\n * Called only from dequeue()\r\n *\r\n * If the ldeqTid is not our own, we must use an HP to protect against\r\n * deqhelp[ldeqTid] being retired-deleted-newed-reenqueued.\r\n */\r\n inline void casDeqAndHead(Node* lhead, Node* lnext, const int tid) {\r\n const int ldeqTid = lnext->deqTid.load();\r\n if (ldeqTid == tid) {\r\n deqhelp[ldeqTid].store(lnext, std::memory_order_release);\r\n } else {\r\n Node* ldeqhelp = hp.protectPtr(kHpDeq, deqhelp[ldeqTid].load(), tid);\r\n if (ldeqhelp != lnext && lhead == head.load()) {\r\n deqhelp[ldeqTid].compare_exchange_strong(ldeqhelp, lnext); // Assign next to request\r\n }\r\n }\r\n head.compare_exchange_strong(lhead, lnext);\r\n }\r\n\r\n\r\n /**\r\n * Called only from dequeue()\r\n *\r\n * Giveup procedure, for when there are no nodes left to dequeue\r\n */\r\n inline void giveUp(Node* myReq, const int tid) {\r\n Node* lhead = head.load();\r\n if (deqhelp[tid].load() != myReq || lhead == tail.load()) return;\r\n hp.protectPtr(kHpHead, lhead, tid);\r\n if (lhead != head.load()) return;\r\n Node* lnext = hp.protectPtr(kHpNext, lhead->next.load(), tid);\r\n if (lhead != head.load()) return;\r\n if (searchNext(lhead, lnext) == IDX_NONE) lnext->casDeqTid(IDX_NONE, tid);\r\n casDeqAndHead(lhead, lnext, tid);\r\n }\r\n\r\npublic:\r\n TurnQueue(int maxThreads=MAX_THREADS) : maxThreads(maxThreads) {\r\n head.store(sentinelNode, std::memory_order_relaxed);\r\n tail.store(sentinelNode, std::memory_order_relaxed);\r\n for (int i = 0; i < maxThreads; i++) {\r\n enqueuers[i].store(nullptr, std::memory_order_relaxed);\r\n // deqself[i] != deqhelp[i] means that isRequest=false\r\n deqself[i].store(new Node(nullptr, 0), std::memory_order_relaxed);\r\n deqhelp[i].store(new Node(nullptr, 0), std::memory_order_relaxed);\r\n }\r\n }\r\n\r\n\r\n ~TurnQueue() {\r\n delete sentinelNode;\r\n while (dequeue(0) != nullptr); // Drain the queue\r\n for (int i=0; i < maxThreads; i++) delete deqself[i].load();\r\n for (int i=0; i < maxThreads; i++) delete deqhelp[i].load();\r\n }\r\n\r\n\r\n static std::string className() { return \"TurnQueue\"; }\r\n\r\n\r\n /**\r\n * Steps when uncontended:\r\n * 1. Add node to enqueuers[]\r\n * 2. Insert node in tail.next using a CAS\r\n * 3. Advance tail to tail.next\r\n * 4. Remove node from enqueuers[]\r\n *\r\n * @param tid The tid must be a UNIQUE index for each thread, in the range 0 to maxThreads-1\r\n */\r\n void enqueue(T* item, const int tid) {\r\n if (item == nullptr) throw std::invalid_argument(\"item can not be nullptr\");\r\n Node* myNode = new Node(item,tid);\r\n enqueuers[tid].store(myNode);\r\n for (int i = 0; i < maxThreads; i++) {\r\n if (enqueuers[tid].load() == nullptr) {\r\n hp.clear(tid);\r\n return; // Some thread did all the steps\r\n }\r\n Node* ltail = hp.protectPtr(kHpTail, tail.load(), tid);\r\n if (ltail != tail.load()) continue; // If the tail advanced maxThreads times, then my node has been enqueued\r\n if (enqueuers[ltail->enqTid].load() == ltail) { // Help a thread do step 4\r\n Node* tmp = ltail;\r\n enqueuers[ltail->enqTid].compare_exchange_strong(tmp, nullptr);\r\n }\r\n for (int j = 1; j < maxThreads+1; j++) { // Help a thread do step 2\r\n Node* nodeToHelp = enqueuers[(j + ltail->enqTid) % maxThreads].load();\r\n if (nodeToHelp == nullptr) continue;\r\n Node* nodenull = nullptr;\r\n ltail->next.compare_exchange_strong(nodenull, nodeToHelp);\r\n break;\r\n }\r\n Node* lnext = ltail->next.load();\r\n \t if (lnext != nullptr) tail.compare_exchange_strong(ltail, lnext); // Help a thread do step 3\r\n }\r\n enqueuers[tid].store(nullptr, std::memory_order_release); // Do step 4, just in case it's not done\r\n hp.clear(tid);\r\n }\r\n\r\n\r\n /**\r\n * Steps when uncontended:\r\n * 1. Publish request to dequeue in dequeuers[tid];\r\n * 2. CAS node->deqTid from IDX_START to tid;\r\n * 3. Set dequeuers[tid] to the newly owned node;\r\n * 4. Advance the head with casHead();\r\n *\r\n * We must protect either head or tail with HP before doing the check for\r\n * empty queue, otherwise we may get into retired-deleted-newed-reenqueued.\r\n *\r\n * @param tid: The tid must be a UNIQUE index for each thread, in the range 0 to maxThreads-1\r\n */\r\n T* dequeue(const int tid) {\r\n Node* prReq = deqself[tid].load(); // Previous request\r\n Node* myReq = deqhelp[tid].load();\r\n deqself[tid].store(myReq); // Step 1\r\n for (int i=0; i < maxThreads; i++) {\r\n if (deqhelp[tid].load() != myReq) break; // No need for HP\r\n Node* lhead = hp.protectPtr(kHpHead, head.load(), tid);\r\n if (lhead != head.load()) continue;\r\n if (lhead == tail.load()) { // Give up\r\n deqself[tid].store(prReq); // Rollback request to dequeue\r\n giveUp(myReq, tid);\r\n if (deqhelp[tid].load() != myReq) {\r\n deqself[tid].store(myReq, std::memory_order_relaxed);\r\n break;\r\n }\r\n hp.clear(tid);\r\n return nullptr;\r\n }\r\n Node* lnext = hp.protectPtr(kHpNext, lhead->next.load(), tid);\r\n if (lhead != head.load()) continue;\r\n \t\t if (searchNext(lhead, lnext) != IDX_NONE) casDeqAndHead(lhead, lnext, tid);\r\n }\r\n Node* myNode = deqhelp[tid].load();\r\n Node* lhead = hp.protectPtr(kHpHead, head.load(), tid); // Do step 4 if needed\r\n if (lhead == head.load() && myNode == lhead->next.load()) head.compare_exchange_strong(lhead, myNode);\r\n hp.clear(tid);\r\n hp.retire(prReq, tid);\r\n return myNode->item;\r\n }\r\n\r\n};\r\n\r\n#endif /* _CR_TURN_QUEUE_HP_H_ */\r\n" }, { "path": "datastructures/sequential/HashSet.hpp", "content": "/******************************************************************************\r\n * Copyright (c) 2014-2018, Pedro Ramalhete, Andreia Correia\r\n * All rights reserved.\r\n *\r\n * Redistribution and use in source and binary forms, with or without\r\n * modification, are permitted provided that the following conditions are met:\r\n * * Redistributions of source code must retain the above copyright\r\n * notice, this list of conditions and the following disclaimer.\r\n * * Redistributions in binary form must reproduce the above copyright\r\n * notice, this list of conditions and the following disclaimer in the\r\n * documentation and/or other materials provided with the distribution.\r\n * * Neither the name of Concurrency Freaks nor the\r\n * names of its contributors may be used to endorse or promote products\r\n * derived from this software without specific prior written permission.\r\n\r\n *\r\n * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\r\n * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED\r\n * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\r\n * DISCLAIMED. IN NO EVENT SHALL BE LIABLE FOR ANY\r\n * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\r\n * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\r\n * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND\r\n * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\r\n * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\r\n * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\r\n ******************************************************************************\r\n */\r\n\r\n#ifndef _UC_HASH_SET_H_\r\n#define _UC_HASH_SET_H_\r\n\r\n#include \r\n#include \r\n#include \r\n\r\n// TODO: change CKey* to CKey&\r\n\r\n// This is a wrapper to std::set, which should be a Red-Black tree\r\ntemplate\r\nclass HashSet {\r\n\r\nprivate:\r\n std::unordered_set set;\r\n\r\npublic:\r\n\r\n static std::string className() { return \"HashSet\"; }\r\n\r\n\r\n bool add(CKey key) {\r\n if (set.find(key) == set.end()) {\r\n set.insert(key); // TODO: can we improve this so we don't have to lookup twice?\r\n return true;\r\n }\r\n return false;\r\n }\r\n\r\n bool remove(CKey key) {\r\n auto iter = set.find(key);\r\n if (iter == set.end()) return false;\r\n set.erase(iter);\r\n return true;\r\n }\r\n\r\n bool contains(CKey key) {\r\n if (set.find(key) == set.end()) return false;\r\n return true; // TODO: optimize this\r\n }\r\n\r\n bool iterateAll(std::function itfun) {\r\n\t\tfor (auto it = set.begin(); it != set.end(); ++it) {\r\n\t\t\tCKey key = *it;\r\n\t\t\tif (!itfun(&key)) return false;\r\n\t\t}\r\n\t\treturn true;\r\n }\r\n\r\n};\r\n\r\n#endif /* _UC_HASH_SET_H_ */\r\n" }, { "path": "datastructures/sequential/LinkedListQueue.hpp", "content": "/******************************************************************************\r\n * Copyright (c) 2014-2017, Pedro Ramalhete, Andreia Correia\r\n * All rights reserved.\r\n *\r\n * Redistribution and use in source and binary forms, with or without\r\n * modification, are permitted provided that the following conditions are met:\r\n * * Redistributions of source code must retain the above copyright\r\n * notice, this list of conditions and the following disclaimer.\r\n * * Redistributions in binary form must reproduce the above copyright\r\n * notice, this list of conditions and the following disclaimer in the\r\n * documentation and/or other materials provided with the distribution.\r\n * * Neither the name of Concurrency Freaks nor the\r\n * names of its contributors may be used to endorse or promote products\r\n * derived from this software without specific prior written permission.\r\n *\r\n * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\r\n * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED\r\n * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\r\n * DISCLAIMED. IN NO EVENT SHALL BE LIABLE FOR ANY\r\n * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\r\n * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\r\n * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND\r\n * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\r\n * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\r\n * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\r\n ******************************************************************************\r\n */\r\n\r\n#ifndef _SEQUENTIAL_LINKED_LIST_QUEUE_H_\r\n#define _SEQUENTIAL_LINKED_LIST_QUEUE_H_\r\n\r\n/**\r\n *

A sequential implementation of Linked List Queue

\r\n *\r\n * This is meant to be used by the Universal Constructs\r\n *\r\n * @author Pedro Ramalhete\r\n * @author Andreia Correia\r\n */\r\ntemplate\r\nclass LinkedListQueue {\r\n\r\nprivate:\r\n struct Node {\r\n T* item;\r\n Node* next {nullptr};\r\n Node(T* userItem) : item{userItem} { }\r\n };\r\n\r\n Node* head {nullptr};\r\n Node* tail {nullptr};\r\n\r\n\r\npublic:\r\n LinkedListQueue(unsigned int maxThreads=0) {\r\n Node* sentinelNode = new Node(nullptr);\r\n head = sentinelNode;\r\n tail = sentinelNode;\r\n }\r\n\r\n\r\n // Universal Constructs need a copy constructor on the underlying data structure\r\n LinkedListQueue(const LinkedListQueue& other) {\r\n head = new Node(nullptr);\r\n Node* node = head;\r\n Node* onode = other.head->next;\r\n while (onode != nullptr) {\r\n node->next = new Node(onode->item);\r\n node = node->next;\r\n onode = onode->next;\r\n }\r\n tail = node;\r\n }\r\n\r\n\r\n ~LinkedListQueue() {\r\n while (dequeue(0) != nullptr); // Drain the queue\r\n Node* lhead = head;\r\n delete lhead;\r\n }\r\n\r\n\r\n static std::string className() { return \"LinkedListQueue\"; }\r\n\r\n\r\n bool enqueue(T* item, const int tid=0) {\r\n if (item == nullptr) return false;\r\n Node* newNode = new Node(item);\r\n tail->next = newNode;\r\n tail = newNode;\r\n return true;\r\n }\r\n\r\n\r\n T* dequeue(const int tid=0) {\r\n Node* lhead = head;\r\n if (lhead == tail) return nullptr;\r\n head = lhead->next;\r\n delete lhead;\r\n return head->item;\r\n }\r\n};\r\n\r\n#endif /* _SEQUENTIAL_LINKED_LIST_QUEUE_H_ */\r\n" }, { "path": "datastructures/sequential/LinkedListSet.hpp", "content": "#ifndef _SEQUENTIAL_LINKED_LIST_SET_H_\r\n#define _SEQUENTIAL_LINKED_LIST_SET_H_\r\n\r\n#include \r\n\r\n/**\r\n *

A sequential implementation of La inked List Set

\r\n *\r\n * This is meant to be used by the Universal Constructs\r\n *\r\n */\r\ntemplate\r\nclass LinkedListSet {\r\n\r\nprivate:\r\n\r\n struct Node {\r\n K key;\r\n Node* next{nullptr};\r\n Node(const K& key) : key{key}, next{nullptr} { }\r\n Node(){ }\r\n };\r\n\r\n Node* head {nullptr};\r\n Node* tail {nullptr};\r\n\r\n\r\npublic:\r\n LinkedListSet() {\r\n\t\tNode* lhead = new Node();\r\n\t\tNode* ltail = new Node();\r\n\t\thead = lhead;\r\n\t\thead->next = ltail;\r\n\t\ttail = ltail;\r\n\r\n }\r\n\r\n\r\n // Universal Constructs need a copy constructor on the underlying data structure\r\n LinkedListSet(const LinkedListSet& other) {\r\n head = new Node();\r\n Node* node = head;\r\n Node* onode = other.head->next;\r\n while (onode != other.tail) {\r\n node->next = new Node(onode->key);\r\n node = node->next;\r\n onode = onode->next;\r\n }\r\n tail = new Node();\r\n node->next = tail;\r\n }\r\n\r\n\r\n ~LinkedListSet() {\r\n\t\t// Delete all the nodes in the list\r\n\t\tNode* prev = head;\r\n\t\tNode* node = prev->next;\r\n\t\twhile (node != tail) {\r\n\t\t\tdelete prev;\r\n\t\t\tprev = node;\r\n\t\t\tnode = node->next;\r\n\t\t}\r\n\t\tdelete prev;\r\n\t\tdelete tail;\r\n }\r\n\r\n\r\n static std::string className() { return \"LinkedListSet\"; }\r\n\r\n\r\n /*\r\n * Adds a node with a key, returns false if the key is already in the set\r\n */\r\n bool add(const K& key) {\r\n Node *prev, *node;\r\n find(key, prev, node);\r\n bool retval = !(node != tail && key == node->key);\r\n if (!retval) return retval;\r\n Node* newNode = new Node(key);\r\n prev->next = newNode;\r\n newNode->next = node;\r\n return retval;\r\n }\r\n\r\n\r\n /*\r\n * Removes a node with an key, returns false if the key is not in the set\r\n */\r\n bool remove(const K& key) {\r\n Node *prev, *node;\r\n find(key, prev, node);\r\n bool retval = (node != tail && key == node->key);\r\n if (!retval) return retval;\r\n prev->next = node->next;\r\n delete node;\r\n return retval;\r\n }\r\n\r\n\r\n /*\r\n * Returns true if it finds a node with a matching key\r\n */\r\n bool contains(const K& key) {\r\n Node *prev, *node;\r\n find(key, prev, node);\r\n return (node != tail && key == node->key);\r\n }\r\n\r\n void find(const K& key, Node*& prev, Node*& node) {\r\n for (prev = head; (node = prev->next) != tail; prev = node){\r\n if ( !(node->key < key) ) break;\r\n }\r\n }\r\n\r\n // Used only for benchmarks\r\n bool addAll(K** keys, const int size) {\r\n bool retval = false;\r\n for (int i = 0; i < size; i++) {\r\n Node *prev, *node;\r\n find(*keys[i], prev, node);\r\n retval = !(node != tail && *keys[i] == node->key);\r\n if (retval) {\r\n Node* newNode = new Node(*keys[i]);\r\n prev->next = newNode;\r\n newNode->next = node;\r\n }\r\n }\r\n return true;\r\n }\r\n};\r\n\r\n#endif /* _SEQUENTIAL_LINKED_LIST_SET_H_ */\r\n" }, { "path": "datastructures/sequential/RedBlackBST.hpp", "content": "#ifndef _RED_BLACK_BST_H_\r\n#define _RED_BLACK_BST_H_\r\n\r\n#include \r\n#include \r\n#include \r\n\r\n// Single-threaded implementation of a Red-Black Tree Map\r\n//http://algs4.cs.princeton.edu/code/edu/princeton/cs/algs4/RedBlackBST.java\r\ntemplate\r\nclass RedBlackBST {\r\n\r\n\r\n struct Node {\r\n K* key;\r\n V* val;\r\n Node* left {nullptr};\r\n Node* right {nullptr};\r\n bool color; // color of parent link\r\n int size; // subtree count\r\n Node(K* key, V* val, bool color, int size) : key{key}, val{val}, color{color}, size{size} {}\r\n };\r\n\r\n Node *root {nullptr}; // root of the BST\r\n\r\n\r\n static const bool RED = true;\r\n static const bool BLACK = false;\r\n\r\npublic:\r\n /**\r\n * Initializes an empty symbol table.\r\n */\r\n RedBlackBST(unsigned int maxThreads=128) { }\r\n\r\n /***************************************************************************\r\n * Node helper methods.\r\n ***************************************************************************/\r\n // is node x red; false if x is null ?\r\n bool isRed(Node* x) {\r\n if (x == nullptr) return false;\r\n return x->color == RED;\r\n }\r\n\r\n // number of node in subtree rooted at x; 0 if x is null\r\n int size(Node* x) {\r\n if (x == nullptr) return 0;\r\n return x->size;\r\n }\r\n\r\n\r\n /**\r\n * Returns the number of key-value pairs in this symbol table.\r\n * @return the number of key-value pairs in this symbol table\r\n */\r\n int size() {\r\n return size(root);\r\n }\r\n\r\n /**\r\n * Is this symbol table empty?\r\n * @return {@code true} if this symbol table is empty and {@code false} otherwise\r\n */\r\n bool isEmpty() {\r\n return root == nullptr;\r\n }\r\n\r\n\r\n /***************************************************************************\r\n * Standard BST search->\r\n ***************************************************************************/\r\n\r\n /**\r\n * Returns the value associated with the given key.\r\n * @param key the key\r\n * @return the value associated with the given key if the key is in the symbol table\r\n * and {@code null} if the key is not in the symbol table\r\n * @throws IllegalArgumentException if {@code key} is {@code null}\r\n */\r\n V* get(K* key) {\r\n if (key == nullptr) throw std::invalid_argument(\"item can not be nullptr\");\r\n return get(root, key);\r\n }\r\n\r\n // value associated with the given key in subtree rooted at x; null if no such key\r\n V* get(Node* x, K* key) {\r\n while (x != nullptr) {\r\n if (*key < *x->key) x = x->left;\r\n else if (*x->key < *key) x = x->right;\r\n else return x->val;\r\n }\r\n return nullptr;\r\n }\r\n\r\n /**\r\n * Does this symbol table contain the given key?\r\n * @param key the key\r\n * @return {@code true} if this symbol table contains {@code key} and\r\n * {@code false} otherwise\r\n * @throws IllegalArgumentException if {@code key} is {@code null}\r\n */\r\n bool contains(K* key) {\r\n return get(key) != nullptr;\r\n }\r\n\r\n /***************************************************************************\r\n * Red-black tree insertion.\r\n ***************************************************************************/\r\n\r\n /**\r\n * Inserts the specified key-value pair into the symbol table, overwriting the old\r\n * value with the new value if the symbol table already contains the specified key.\r\n * Deletes the specified key (and its associated value) from this symbol table\r\n * if the specified value is {@code null}.\r\n *\r\n * @param key the key\r\n * @param val the value\r\n * @throws IllegalArgumentException if {@code key} is {@code null}\r\n */\r\n void put(K* key, V* val) {\r\n if (key == nullptr) throw std::invalid_argument(\"item can not be nullptr\");\r\n if (val == nullptr) {\r\n deleteKey(key);\r\n return;\r\n }\r\n\r\n root = put(root, key, val);\r\n root->color = BLACK;\r\n // assert check();\r\n }\r\n\r\n // insert the key-value pair in the subtree rooted at h\r\n Node* put(Node* h, K* key, V* val) {\r\n if (h == nullptr) return new Node(key, val, RED, 1);\r\n if (*key < *h->key) h->left = put(h->left, key, val);\r\n else if (*h->key < *key) h->right = put(h->right, key, val);\r\n else h->val = val;\r\n\r\n // fix-up any right-leaning links\r\n if (isRed(h->right) && !isRed(h->left)) h = rotateLeft(h);\r\n if (isRed(h->left) && isRed(h->left->left)) h = rotateRight(h);\r\n if (isRed(h->left) && isRed(h->right)) flipColors(h);\r\n h->size = size(h->left) + size(h->right) + 1;\r\n\r\n return h;\r\n }\r\n\r\n /***************************************************************************\r\n * Red-black tree deletion.\r\n ***************************************************************************/\r\n\r\n /**\r\n * Removes the smallest key and associated value from the symbol table.\r\n * @throws NoSuchElementException if the symbol table is empty\r\n */\r\n void deleteMin() {\r\n if (isEmpty()) throw std::invalid_argument(\"item can not be nullptr\");\r\n\r\n // if both children of root are black, set root to red\r\n if (!isRed(root->left) && !isRed(root->right))\r\n root->color = RED;\r\n\r\n root = deleteMin(root);\r\n if (!isEmpty()) root->color = BLACK;\r\n // assert check();\r\n }\r\n\r\n // delete the key-value pair with the minimum key rooted at h\r\n Node* deleteMin(Node* h) {\r\n if (h->left == nullptr)\r\n return nullptr;\r\n\r\n if (!isRed(h->left) && !isRed(h->left->left))\r\n h = moveRedLeft(h);\r\n\r\n h->left = deleteMin(h->left);\r\n return balance(h);\r\n }\r\n\r\n\r\n /**\r\n * Removes the largest key and associated value from the symbol table.\r\n * @throws NoSuchElementException if the symbol table is empty\r\n */\r\n void deleteMax() {\r\n if (isEmpty()) throw std::invalid_argument(\"item can not be nullptr\");\r\n\r\n // if both children of root are black, set root to red\r\n if (!isRed(root->left) && !isRed(root->right))\r\n root->color = RED;\r\n\r\n root = deleteMax(root);\r\n if (!isEmpty()) root->color = BLACK;\r\n // assert check();\r\n }\r\n\r\n // delete the key-value pair with the maximum key rooted at h\r\n Node* deleteMax(Node* h) {\r\n if (isRed(h->left))\r\n h = rotateRight(h);\r\n\r\n if (h->right == nullptr)\r\n return nullptr;\r\n\r\n if (!isRed(h->right) && !isRed(h->right->left))\r\n h = moveRedRight(h);\r\n\r\n h->right = deleteMax(h->right);\r\n\r\n return balance(h);\r\n }\r\n\r\n /**\r\n * Removes the specified key and its associated value from this symbol table\r\n * (if the key is in this symbol table).\r\n *\r\n * @param key the key\r\n * @throws IllegalArgumentException if {@code key} is {@code null}\r\n */\r\n void deleteKey(K* key) {\r\n if (key == nullptr) throw std::invalid_argument(\"item can not be nullptr\");\r\n if (!contains(key)) return;\r\n\r\n // if both children of root are black, set root to red\r\n if (!isRed(root->left) && !isRed(root->right))\r\n root->color = RED;\r\n\r\n root = deleteKey(root, key);\r\n if (!isEmpty()) root->color = BLACK;\r\n // assert check();\r\n }\r\n\r\n // delete the key-value pair with the given key rooted at h\r\n Node* deleteKey(Node* h, K* key) {\r\n // assert get(h, key) != null;\r\n\r\n if (*key < *h->key) {\r\n if (!isRed(h->left) && !isRed(h->left->left))\r\n h = moveRedLeft(h);\r\n h->left = deleteKey(h->left, key);\r\n }\r\n else {\r\n if (isRed(h->left))\r\n h = rotateRight(h);\r\n if (*key == *h->key && (h->right == nullptr))\r\n return nullptr;\r\n if (!isRed(h->right) && !isRed(h->right->left))\r\n h = moveRedRight(h);\r\n if (*key == *h->key) {\r\n Node* x = min(h->right);\r\n h->key = x->key;\r\n h->val = x->val;\r\n // h->val = get(h->right, min(h->right).key);\r\n // h->key = min(h->right).key;\r\n h->right = deleteMin(h->right);\r\n }\r\n else h->right = deleteKey(h->right, key);\r\n }\r\n return balance(h);\r\n }\r\n\r\n /***************************************************************************\r\n * Red-black tree helper functions.\r\n ***************************************************************************/\r\n\r\n // make a left-leaning link lean to the right\r\n Node* rotateRight(Node* h) {\r\n // assert (h != null) && isRed(h->left);\r\n Node* x = h->left;\r\n h->left = x->right;\r\n x->right = h;\r\n x->color = x->right->color;\r\n x->right->color = RED;\r\n x->size = h->size;\r\n h->size = size(h->left) + size(h->right) + 1;\r\n return x;\r\n }\r\n\r\n // make a right-leaning link lean to the left\r\n Node* rotateLeft(Node* h) {\r\n // assert (h != null) && isRed(h->right);\r\n Node* x = h->right;\r\n h->right = x->left;\r\n x->left = h;\r\n x->color = x->left->color;\r\n x->left->color = RED;\r\n x->size = h->size;\r\n h->size = size(h->left) + size(h->right) + 1;\r\n return x;\r\n }\r\n\r\n // flip the colors of a node and its two children\r\n void flipColors(Node* h) {\r\n // h must have opposite color of its two children\r\n // assert (h != null) && (h->left != null) && (h->right != null);\r\n // assert (!isRed(h) && isRed(h->left) && isRed(h->right))\r\n // || (isRed(h) && !isRed(h->left) && !isRed(h->right));\r\n h->color = !h->color;\r\n h->left->color = !h->left->color;\r\n h->right->color = !h->right->color;\r\n }\r\n\r\n // Assuming that h is red and both h->left and h->left.left\r\n // are black, make h->left or one of its children red.\r\n Node* moveRedLeft(Node* h) {\r\n // assert (h != null);\r\n // assert isRed(h) && !isRed(h->left) && !isRed(h->left.left);\r\n\r\n flipColors(h);\r\n if (isRed(h->right->left)) {\r\n h->right = rotateRight(h->right);\r\n h = rotateLeft(h);\r\n flipColors(h);\r\n }\r\n return h;\r\n }\r\n\r\n // Assuming that h is red and both h->right and h->right.left\r\n // are black, make h->right or one of its children red.\r\n Node* moveRedRight(Node* h) {\r\n // assert (h != null);\r\n // assert isRed(h) && !isRed(h->right) && !isRed(h->right.left);\r\n flipColors(h);\r\n if (isRed(h->left->left)) {\r\n h = rotateRight(h);\r\n flipColors(h);\r\n }\r\n return h;\r\n }\r\n\r\n // restore red-black tree invariant\r\n Node* balance(Node* h) {\r\n // assert (h != null);\r\n\r\n if (isRed(h->right)) h = rotateLeft(h);\r\n if (isRed(h->left) && isRed(h->left->left)) h = rotateRight(h);\r\n if (isRed(h->left) && isRed(h->right)) flipColors(h);\r\n\r\n h->size = size(h->left) + size(h->right) + 1;\r\n return h;\r\n }\r\n\r\n\r\n /***************************************************************************\r\n * Utility functions.\r\n ***************************************************************************/\r\n\r\n /**\r\n * Returns the height of the BST (for debugging).\r\n * @return the height of the BST (a 1-node tree has height 0)\r\n */\r\n int height() {\r\n return height(root);\r\n }\r\n int height(Node* x) {\r\n if (x == nullptr) return -1;\r\n return 1 + std::max(height(x->left), height(x->right));\r\n }\r\n\r\n /***************************************************************************\r\n * Ordered symbol table methods.\r\n ***************************************************************************/\r\n\r\n /**\r\n * Returns the smallest key in the symbol table.\r\n * @return the smallest key in the symbol table\r\n * @throws NoSuchElementException if the symbol table is empty\r\n */\r\n K* min() {\r\n if (isEmpty()) throw std::invalid_argument(\"item can not be nullptr\");\r\n return min(root).key;\r\n }\r\n\r\n // the smallest key in subtree rooted at x; null if no such key\r\n Node* min(Node* x) {\r\n // assert x != null;\r\n if (x->left == nullptr) return x;\r\n else return min(x->left);\r\n }\r\n\r\n /**\r\n * Returns the largest key in the symbol table.\r\n * @return the largest key in the symbol table\r\n * @throws NoSuchElementException if the symbol table is empty\r\n */\r\n K* max() {\r\n if (isEmpty()) throw std::invalid_argument(\"item can not be nullptr\");\r\n return max(root).key;\r\n }\r\n\r\n // the largest key in the subtree rooted at x; null if no such key\r\n Node* max(Node* x) {\r\n // assert x != null;\r\n if (x->right == nullptr) return x;\r\n else return max(x->right);\r\n }\r\n\r\n\r\n /**\r\n * Returns the largest key in the symbol table less than or equal to {@code key}.\r\n * @param key the key\r\n * @return the largest key in the symbol table less than or equal to {@code key}\r\n * @throws NoSuchElementException if there is no such key\r\n * @throws IllegalArgumentException if {@code key} is {@code null}\r\n */\r\n K* floor(K* key) {\r\n if (key == nullptr) throw std::invalid_argument(\"item can not be nullptr\");\r\n if (isEmpty()) throw std::invalid_argument(\"item can not be nullptr\");\r\n Node* x = floor(root, key);\r\n if (x == nullptr) return nullptr;\r\n else return x->key;\r\n }\r\n\r\n // the largest key in the subtree rooted at x less than or equal to the given key\r\n Node* floor(Node* x, K* key) {\r\n if (x == nullptr) return nullptr;\r\n if (*key == *x->key) return x;\r\n if (*key < *x->key) return floor(x->left, key);\r\n Node* t = floor(x->right, key);\r\n if (t != nullptr) return t;\r\n else return x;\r\n }\r\n\r\n /**\r\n * Returns the smallest key in the symbol table greater than or equal to {@code key}.\r\n * @param key the key\r\n * @return the smallest key in the symbol table greater than or equal to {@code key}\r\n * @throws NoSuchElementException if there is no such key\r\n * @throws IllegalArgumentException if {@code key} is {@code null}\r\n */\r\n K* ceiling(K* key) {\r\n if (key == nullptr) throw std::invalid_argument(\"item can not be nullptr\");\r\n if (isEmpty()) throw std::invalid_argument(\"item can not be nullptr\");\r\n Node* x = ceiling(root, key);\r\n if (x == nullptr) return nullptr;\r\n else return x->key;\r\n }\r\n\r\n // the smallest key in the subtree rooted at x greater than or equal to the given key\r\n Node* ceiling(Node* x, K* key) {\r\n if (x == nullptr) return nullptr;\r\n int cmp = key.compareTo(x->key);\r\n if (*key == *x->key) return x;\r\n if (*x->key < *key) return ceiling(x->right, key);\r\n Node* t = ceiling(x->left, key);\r\n if (t != nullptr) return t;\r\n else return x;\r\n }\r\n\r\n /**\r\n * Return the kth smallest key in the symbol table.\r\n * @param k the order statistic\r\n * @return the {@code k}th smallest key in the symbol table\r\n * @throws IllegalArgumentException unless {@code k} is between 0 and\r\n * n1\r\n */\r\n K* select(int k) {\r\n if (k < 0 || k >= size()) {\r\n throw std::invalid_argument(\"item can not be nullptr\");\r\n }\r\n Node x = select(root, k);\r\n return x->key;\r\n }\r\n\r\n // the key of rank k in the subtree rooted at x\r\n Node* select(Node* x, int k) {\r\n // assert x != null;\r\n // assert k >= 0 && k < size(x);\r\n int t = size(x->left);\r\n if (t > k) return select(x->left, k);\r\n else if (t < k) return select(x->right, k-t-1);\r\n else return x;\r\n }\r\n\r\n /**\r\n * Return the number of keys in the symbol table strictly less than {@code key}.\r\n * @param key the key\r\n * @return the number of keys in the symbol table strictly less than {@code key}\r\n * @throws IllegalArgumentException if {@code key} is {@code null}\r\n */\r\n int rank(K* key) {\r\n if (key == nullptr) throw std::invalid_argument(\"item can not be nullptr\");\r\n return rank(key, root);\r\n }\r\n\r\n // number of keys less than key in the subtree rooted at x\r\n int rank(K* key, Node* x) {\r\n if (x == nullptr) return 0;\r\n if (*key < *x->key) return rank(key, x->left);\r\n else if (*x->key < *key) return 1 + size(x->left) + rank(key, x->right);\r\n else return size(x->left);\r\n }\r\n\r\n /***************************************************************************\r\n * Range count and range search->\r\n ***************************************************************************/\r\n\r\n\r\n /**\r\n * Returns the number of keys in the symbol table in the given range.\r\n *\r\n * @param lo minimum endpoint\r\n * @param hi maximum endpoint\r\n * @return the number of keys in the sybol table between {@code lo}\r\n * (inclusive) and {@code hi} (inclusive)\r\n * @throws IllegalArgumentException if either {@code lo} or {@code hi}\r\n * is {@code null}\r\n */\r\n int size(K* lo, K* hi) {\r\n if (lo == nullptr) throw std::invalid_argument(\"item can not be nullptr\");\r\n if (hi == nullptr) throw std::invalid_argument(\"item can not be nullptr\");\r\n\r\n if (*lo < *hi) return 0;\r\n if (contains(hi)) return rank(hi) - rank(lo) + 1;\r\n else return rank(hi) - rank(lo);\r\n }\r\n\r\n\r\n /***************************************************************************\r\n * Check integrity of red-black tree data structure.\r\n ***************************************************************************/\r\n bool check() {\r\n if (!isBST()) std::cout << \"Not in symmetric order\\n\";\r\n if (!isSizeConsistent()) std::cout << \"Subtree counts not consistent\\n\";\r\n //if (!isRankConsistent()) std::cout << \"Ranks not consistent\\n\";\r\n if (!is23()) std::cout << \"Not a 2-3 tree\\n\";\r\n if (!isBalanced()) std::cout << \"Not balanced\\n\";\r\n return isBST() && isSizeConsistent() && is23() && isBalanced();\r\n }\r\n\r\n // does this binary tree satisfy symmetric order?\r\n // Note: this test also ensures that data structure is a binary tree since order is strict\r\n bool isBST() {\r\n return isBST(root, nullptr, nullptr);\r\n }\r\n\r\n // is the tree rooted at x a BST with all keys strictly between min and max\r\n // (if min or max is null, treat as empty constraint)\r\n // Credit: Bob Dondero's elegant solution\r\n bool isBST(Node* x, K* min, K* max) {\r\n if (x == nullptr) return true;\r\n if (min != nullptr && x->key.compareTo(min) <= 0) return false;\r\n if (max != nullptr && x->key.compareTo(max) >= 0) return false;\r\n return isBST(x->left, min, x->key) && isBST(x->right, x->key, max);\r\n }\r\n\r\n // are the size fields correct?\r\n bool isSizeConsistent() { return isSizeConsistent(root); }\r\n bool isSizeConsistent(Node* x) {\r\n if (x == nullptr) return true;\r\n if (x->size != size(x->left) + size(x->right) + 1) return false;\r\n return isSizeConsistent(x->left) && isSizeConsistent(x->right);\r\n }\r\n\r\n /*\r\n // check that ranks are consistent\r\n bool isRankConsistent() {\r\n for (int i = 0; i < size(); i++)\r\n if (i != rank(select(i))) return false;\r\n for (K* key : keys())\r\n if (key.compareTo(select(rank(key))) != 0) return false;\r\n return true;\r\n }\r\n */\r\n\r\n // Does the tree have no red right links, and at most one (left)\r\n // red links in a row on any path?\r\n bool is23() { return is23(root); }\r\n bool is23(Node* x) {\r\n if (x == nullptr) return true;\r\n if (isRed(x->right)) return false;\r\n if (x != root && isRed(x) && isRed(x->left))\r\n return false;\r\n return is23(x->left) && is23(x->right);\r\n }\r\n\r\n // do all paths from root to leaf have same number of black edges?\r\n bool isBalanced() {\r\n int black = 0; // number of black links on path from root to min\r\n Node x = root;\r\n while (x != nullptr) {\r\n if (!isRed(x)) black++;\r\n x = x->left;\r\n }\r\n return isBalanced(root, black);\r\n }\r\n\r\n // does every path from the root to a leaf have the given number of black links?\r\n bool isBalanced(Node* x, int black) {\r\n if (x == nullptr) return black == 0;\r\n if (!isRed(x)) black--;\r\n return isBalanced(x->left, black) && isBalanced(x->right, black);\r\n }\r\n\r\n\r\n\r\n // Set methods\r\n bool add(K* key, const int tid) {\r\n if (contains(key)) return false;\r\n put(key,key);\r\n return true;\r\n }\r\n\r\n bool remove(K* key, const int tid) {\r\n if (!contains(key)) return false;\r\n deleteKey(key);\r\n return true;\r\n }\r\n\r\n inline bool contains(K* key, const int tid) {\r\n return contains(key);\r\n }\r\n\r\n std::string className() { return \"RedBlackBST\"; }\r\n\r\n};\r\n\r\n#endif // _RED_BLACK_BST_H_\r\n" }, { "path": "datastructures/sequential/SortedArraySet.hpp", "content": "/******************************************************************************\r\n * Copyright (c) 2014-2016, Pedro Ramalhete, Andreia Correia\r\n * All rights reserved.\r\n *\r\n * Redistribution and use in source and binary forms, with or without\r\n * modification, are permitted provided that the following conditions are met:\r\n * * Redistributions of source code must retain the above copyright\r\n * notice, this list of conditions and the following disclaimer.\r\n * * Redistributions in binary form must reproduce the above copyright\r\n * notice, this list of conditions and the following disclaimer in the\r\n * documentation and/or other materials provided with the distribution.\r\n * * Neither the name of Concurrency Freaks nor the\r\n * names of its contributors may be used to endorse or promote products\r\n * derived from this software without specific prior written permission.\r\n\r\n *\r\n * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\r\n * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED\r\n * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\r\n * DISCLAIMED. IN NO EVENT SHALL BE LIABLE FOR ANY\r\n * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\r\n * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\r\n * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND\r\n * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\r\n * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\r\n * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\r\n ******************************************************************************\r\n */\r\n\r\n#ifndef _SORTEDARRAYSET_H_\r\n#define _SORTEDARRAYSET_H_\r\n\r\n#include \r\n\r\n// TODO: Test this for correctness\r\n\r\n/**\r\n * This is storing the pointers to the T instances, not the actual T instances.\r\n */\r\ntemplate\r\nclass SortedArraySet {\r\n\r\nprivate:\r\n unsigned max_size = 32;\r\n T** vec; // TODO: change this to T if we change the API from T* to T&\r\n unsigned size = 0;\r\n static const int NOT_FOUND = 0;\r\n //std::atomic flag {false}; // For de debugging\r\n\r\n int lookup(T* key) {\r\n // Cover the special case of an empty array\r\n if (size== 0) return NOT_FOUND;\r\n int minPos = 0;\r\n int maxPos = size-1;\r\n //std::cout << \"vec[0] = \" << vec[0] << \"\\n\";\r\n // Special cases for first and last items\r\n if (*key < *(vec[0])) return NOT_FOUND;\r\n if (*key == *(vec[0])) return 0;\r\n if (*key == *(vec[maxPos])) return maxPos;\r\n if (*(vec[maxPos]) < *key) return maxPos+1;\r\n while (true) {\r\n int pos = (maxPos-minPos)/2 + minPos;\r\n\r\n if (*key < *(vec[pos])) {\r\n maxPos = pos;\r\n } else if (*key == *(vec[pos])) {\r\n return pos;\r\n } else {\r\n minPos = pos;\r\n }\r\n if (maxPos-minPos <= 1) {\r\n return maxPos;\r\n }\r\n }\r\n }\r\n\r\n\r\npublic:\r\n SortedArraySet() {\r\n vec = new T*[max_size];\r\n }\r\n\r\n ~SortedArraySet() {\r\n delete[] vec;\r\n }\r\n\r\n // We need a copy constructor to be able to use it in CXMutation\r\n SortedArraySet(const SortedArraySet& fromssv) {\r\n\r\n vec = new T*[fromssv.max_size];\r\n max_size = fromssv.max_size;\r\n size = fromssv.size;\r\n for(unsigned i=0;iseq<<\" \"<tid<<\" vex \"<seq<<\" \"<tid<<\" vex \"<seq<< \" \" << vec[index]->tid<<\"\\n\";\r\n //assert(false);\r\n //flag.store(false);\r\n return false;\r\n }\r\n\r\n if(size+1==max_size){\r\n T** newvec = new T*[2*max_size];\r\n for(unsigned i=0;i=index+1;i--){\r\n vec[i]=vec[i-1];\r\n }\r\n }\r\n vec[index] = key;\r\n }\r\n size++;\r\n //flag.store(false);\r\n return true;\r\n }\r\n\r\n\r\n bool contains(T* key) {\r\n //if (flag.load()) std::cout << \"contains() ERRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRROOOOOOOOOOOOOOR\\n\";\r\n unsigned index = lookup(key);\r\n if (index == size) {\r\n return false;\r\n }\r\n return *key == *(vec[index]);\r\n }\r\n\r\n\r\n bool print() { // For debug purposes\r\n for(unsigned i=0;i BE LIABLE FOR ANY\r\n * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\r\n * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\r\n * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND\r\n * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\r\n * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\r\n * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\r\n ******************************************************************************\r\n */\r\n\r\n#ifndef _SORTED_VECTOR_SET_H_\r\n#define _SORTED_VECTOR_SET_H_\r\n\r\n#include \r\n#include \r\n\r\n// TODO: Test this for correctness\r\n\r\n/**\r\n * This is storing the pointers to the T instances, not the actual T instances.\r\n */\r\ntemplate\r\nclass SortedVectorSet {\r\n\r\nprivate:\r\n std::vector vec; // TODO: change this to T if we change the API from T* to T&\r\n static const int NOT_FOUND = 0;\r\n //std::atomic flag {false}; // For de debugging\r\n\r\n int lookup(T* key) {\r\n // Cover the special case of an empty array\r\n if (vec.size()== 0) return NOT_FOUND;\r\n int minPos = 0;\r\n int maxPos = vec.size()-1;\r\n // Special cases for first and last items\r\n if (*key < *(vec[0])) return NOT_FOUND;\r\n if (*key == *(vec[0])) return 0;\r\n if (*key == *(vec[maxPos])) return maxPos;\r\n if (*(vec[maxPos]) < *key) return maxPos+1;\r\n while (true) {\r\n int pos = (maxPos-minPos)/2 + minPos;\r\n if (*key < *(vec[pos])) {\r\n maxPos = pos;\r\n } else if (*key == *(vec[pos])) {\r\n return pos;\r\n } else {\r\n minPos = pos;\r\n }\r\n if (maxPos-minPos <= 1) {\r\n return maxPos;\r\n }\r\n }\r\n }\r\n\r\n\r\npublic:\r\n SortedVectorSet() { }\r\n\r\n // We need a copy constructor to be able to use it in CXMutation\r\n SortedVectorSet(const SortedVectorSet& from) {\r\n vec = from.vec; // Do a copy of the vector\r\n }\r\n\r\n static std::string className() { return \"SortedVectorSet\"; }\r\n\r\n /**\r\n * When the curr.key is seen to be null it means we reached the tail node\r\n */\r\n bool remove(T* key) {\r\n //if (flag.load()) std::cout << \"remove() ERRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRROOOOOOOOOOOOOOR\\n\";\r\n //flag.store(true);\r\n unsigned index = lookup(key);\r\n if (index == vec.size()) {\r\n //std::cout<<\"remove key \"<seq<<\" \"<tid<<\" vex \"<seq<<\" \"<tid<<\" vex \"<seq<< \" \" << vec[index]->tid<<\"\\n\";\r\n //assert(false);\r\n //flag.store(false);\r\n return false;\r\n }\r\n vec.insert(vec.begin()+index, key);\r\n //flag.store(false);\r\n return true;\r\n }\r\n\r\n\r\n bool contains(T* key) {\r\n //if (flag.load()) std::cout << \"contains() ERRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRROOOOOOOOOOOOOOR\\n\";\r\n unsigned index = lookup(key);\r\n if (index == vec.size()) {\r\n return false;\r\n }\r\n return *key == *(vec[index]);\r\n }\r\n\r\n\r\n bool print() { // For debug purposes\r\n for (T* p : vec) std:: cout << p << \",\";\r\n std::cout << \"\\n\";\r\n return true;\r\n }\r\n};\r\n\r\n#endif /* _SORTED_VECTOR_SET_H_ */\r\n" }, { "path": "datastructures/sequential/TreeSet.hpp", "content": "/******************************************************************************\r\n * Copyright (c) 2014-2016, Pedro Ramalhete, Andreia Correia\r\n * All rights reserved.\r\n *\r\n * Redistribution and use in source and binary forms, with or without\r\n * modification, are permitted provided that the following conditions are met:\r\n * * Redistributions of source code must retain the above copyright\r\n * notice, this list of conditions and the following disclaimer.\r\n * * Redistributions in binary form must reproduce the above copyright\r\n * notice, this list of conditions and the following disclaimer in the\r\n * documentation and/or other materials provided with the distribution.\r\n * * Neither the name of Concurrency Freaks nor the\r\n * names of its contributors may be used to endorse or promote products\r\n * derived from this software without specific prior written permission.\r\n\r\n *\r\n * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\r\n * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED\r\n * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\r\n * DISCLAIMED. IN NO EVENT SHALL BE LIABLE FOR ANY\r\n * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\r\n * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\r\n * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND\r\n * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\r\n * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\r\n * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\r\n ******************************************************************************\r\n */\r\n\r\n#ifndef _CX_TREE_SET_H_\r\n#define _CX_TREE_SET_H_\r\n\r\n#include \r\n#include \r\n#include \r\n\r\n//#include \"../datastructures/sequential/RedBlackBST.hpp\"\r\n\r\n// TODO: change CKey* to CKey&\r\n\r\n// This is a wrapper to std::set, which should be a Red-Black tree\r\ntemplate\r\nclass TreeSet {\r\n\r\nprivate:\r\n std::set set;\r\n // Use this instead if we want to have control over the Red-Black tree\r\n //RedBlackBST set;\r\n\r\npublic:\r\n\r\n static std::string className() { return \"TreeSet\"; }\r\n\r\n\r\n bool add(CKey key) {\r\n if (set.find(key) == set.end()) {\r\n set.insert(key); // TODO: can we improve this so we don't have to lookup twice?\r\n return true;\r\n }\r\n return false;\r\n }\r\n\r\n bool remove(CKey key) {\r\n auto iter = set.find(key);\r\n if (iter == set.end()) return false;\r\n set.erase(iter);\r\n return true;\r\n }\r\n\r\n bool contains(CKey key) {\r\n if (set.find(key) == set.end()) return false;\r\n return true; // TODO: optimize this\r\n }\r\n\r\n bool iterateAll(std::function itfunc) {\r\n\t\tfor (auto it = set.begin(); it != set.end(); ++it) {\r\n\t\t\tCKey key = *it;\r\n\t\t\tif (!itfunc(&key)) return false;\r\n\t\t}\r\n\t\treturn true;\r\n }\r\n/*\r\n bool add(CKey* key) {\r\n return set.add(key, 0);\r\n }\r\n\r\n bool remove(CKey* key) {\r\n return set.remove(key, 0);\r\n }\r\n\r\n bool contains(CKey* key) {\r\n return set.contains(key, 0);\r\n }\r\n*/\r\n};\r\n\r\n#endif /* _TREE_SET_H_ */\r\n" }, { "path": "datastructures/treemaps/ESTMRedBlackTree.hpp", "content": "#ifndef _ESTM_RED_BLACK_BST_H_\r\n#define _ESTM_RED_BLACK_BST_H_\r\n\r\n#include \r\n#include \r\n#include \r\n\r\n#include \"stms/ESTM.hpp\"\r\n\r\n// Adapted from Java to C++ from the original at http://algs4.cs.princeton.edu/code/edu/princeton/cs/algs4/RedBlackBST.java\r\ntemplate\r\nclass ESTMRedBlackTree {\r\n const int64_t COLOR_RED = 0;\r\n const int64_t COLOR_BLACK = 1;\r\n\r\n struct Node {\r\n estm::tmtype key;\r\n estm::tmtype val;\r\n estm::tmtype left {nullptr};\r\n estm::tmtype right {nullptr};\r\n estm::tmtype color; // color of parent link\r\n estm::tmtype size; // subtree count\r\n Node(const K& key, const V& val, int64_t color, int64_t size) : key{key}, val{val}, color{color}, size{size} {}\r\n };\r\n\r\n estm::tmtype root {nullptr}; // root of the BST\r\n\r\n inline void assignAndFreeIfNull(estm::tmtype& z, Node* w) {\r\n Node* tofree = z;\r\n z = w;\r\n if (w == nullptr) estm::tmDelete(tofree);\r\n }\r\n\r\npublic:\r\n /**\r\n * Initializes an empty symbol table.\r\n */\r\n ESTMRedBlackTree(int maxThreads=0){ }\r\n\r\n ~ESTMRedBlackTree() {\r\n // The transaction log is not enough to delete everything if there are too many, so we delete 1000 per transaction\r\n for (int i = 0; i < 1000; i++) {\r\n estm::updateTx([&] () {\r\n if (root == nullptr) return true;\r\n deleteMin();\r\n return true;\r\n });\r\n }\r\n }\r\n\r\n /***************************************************************************\r\n * Node helper methods.\r\n ***************************************************************************/\r\n // is node x red; false if x is null ?\r\n bool isRed(Node* x) {\r\n if (x == nullptr) return false;\r\n return x->color == COLOR_RED;\r\n }\r\n\r\n // number of node in subtree rooted at x; 0 if x is null\r\n int size(Node* x) {\r\n if (x == nullptr) return 0;\r\n return x->size;\r\n }\r\n\r\n\r\n /**\r\n * Returns the number of key-value pairs in this symbol table.\r\n * @return the number of key-value pairs in this symbol table\r\n */\r\n int size() {\r\n return size(root);\r\n }\r\n\r\n /**\r\n * Is this symbol table empty?\r\n * @return {@code true} if this symbol table is empty and {@code false} otherwise\r\n */\r\n bool isEmpty() {\r\n return root == nullptr;\r\n }\r\n\r\n\r\n /***************************************************************************\r\n * Standard BST search->\r\n ***************************************************************************/\r\n\r\n /**\r\n * Returns the value associated with the given key.\r\n * @param key the key\r\n * @return the value associated with the given key if the key is in the symbol table\r\n * and {@code null} if the key is not in the symbol table\r\n * @throws IllegalArgumentException if {@code key} is {@code null}\r\n */\r\n bool innerGet(K& key, V& oldValue, const bool saveOldValue) {\r\n bool found = get(root, key);\r\n if (!found) return false;\r\n //if (saveOldValue) oldValue = *val; // Copy of V\r\n return true;\r\n }\r\n\r\n // value associated with the given key in subtree rooted at x; null if no such key\r\n bool get(Node* x, K& key) {\r\n while (x != nullptr) {\r\n if (key < x->key) x = x->left;\r\n else if (x->key < key) x = x->right;\r\n else return true;\r\n }\r\n return false;\r\n }\r\n\r\n /**\r\n * Does this symbol table contain the given key?\r\n * @param key the key\r\n * @return {@code true} if this symbol table contains {@code key} and\r\n * {@code false} otherwise\r\n * @throws IllegalArgumentException if {@code key} is {@code null}\r\n */\r\n bool containsKey(const K& key) {\r\n return get(key) != nullptr;\r\n }\r\n\r\n /***************************************************************************\r\n * Red-black tree insertion.\r\n ***************************************************************************/\r\n\r\n /**\r\n * Inserts the specified key-value pair into the symbol table, overwriting the old\r\n * value with the new value if the symbol table already contains the specified key.\r\n * Deletes the specified key (and its associated value) from this symbol table\r\n * if the specified value is {@code null}.\r\n *\r\n * @param key the key\r\n * @param val the value\r\n * @throws IllegalArgumentException if {@code key} is {@code null}\r\n */\r\n bool innerPut(const K& key, const V& value) {\r\n \tbool ret = false;\r\n root = put(root, key, value, ret);\r\n root->color = COLOR_BLACK;\r\n return ret;\r\n }\r\n\r\n // insert the key-value pair in the subtree rooted at h\r\n Node* put(Node* h, const K& key, const V& val, bool& ret) {\r\n if (h == nullptr) {\r\n \tret = true;\r\n \treturn estm::tmNew(key, val, COLOR_RED, 1);\r\n }\r\n if (key < h->key) h->left = put(h->left, key, val, ret);\r\n else if (h->key < key) h->right = put(h->right, key, val, ret);\r\n else h->val = val;\r\n // fix-up any right-leaning links\r\n if (isRed(h->right) && !isRed(h->left)) h = rotateLeft(h);\r\n if (isRed(h->left) && isRed(h->left->left)) h = rotateRight(h);\r\n if (isRed(h->left) && isRed(h->right)) flipColors(h);\r\n h->size = size(h->left) + size(h->right) + 1;\r\n\r\n return h;\r\n }\r\n\r\n /***************************************************************************\r\n * Red-black tree deletion.\r\n ***************************************************************************/\r\n\r\n /**\r\n * Removes the smallest key and associated value from the symbol table.\r\n * @throws NoSuchElementException if the symbol table is empty\r\n */\r\n void deleteMin() {\r\n if (isEmpty()) return;\r\n // if both children of root are black, set root to red\r\n if (!isRed(root->left) && !isRed(root->right))\r\n root->color = COLOR_RED;\r\n assignAndFreeIfNull(root, deleteMin(root));\r\n if (!isEmpty()) root->color = COLOR_BLACK;\r\n // assert check();\r\n }\r\n\r\n // delete the key-value pair with the minimum key rooted at h\r\n Node* deleteMin(Node* h) {\r\n if (h->left == nullptr)\r\n return nullptr;\r\n if (!isRed(h->left) && !isRed(h->left->left))\r\n h = moveRedLeft(h);\r\n assignAndFreeIfNull(h->left, deleteMin(h->left));\r\n return balance(h);\r\n }\r\n\r\n\r\n /**\r\n * Removes the largest key and associated value from the symbol table.\r\n * @throws NoSuchElementException if the symbol table is empty\r\n */\r\n void deleteMax() {\r\n if (isEmpty()) return;\r\n\r\n // if both children of root are black, set root to red\r\n if (!isRed(root->left) && !isRed(root->right))\r\n root->color = COLOR_RED;\r\n\r\n root = deleteMax(root);\r\n if (!isEmpty()) root->color = COLOR_BLACK;\r\n // assert check();\r\n }\r\n\r\n // delete the key-value pair with the maximum key rooted at h\r\n Node* deleteMax(Node* h) {\r\n if (isRed(h->left))\r\n h = rotateRight(h);\r\n\r\n if (h->right == nullptr)\r\n return nullptr;\r\n\r\n if (!isRed(h->right) && !isRed(h->right->left))\r\n h = moveRedRight(h);\r\n\r\n h->right = deleteMax(h->right);\r\n\r\n return balance(h);\r\n }\r\n\r\n /**\r\n * Removes the specified key and its associated value from this symbol table\r\n * (if the key is in this symbol table).\r\n *\r\n * @param key the key\r\n */\r\n void innerRemove(const K& key) {\r\n // if both children of root are black, set root to red\r\n if (!isRed(root->left) && !isRed(root->right)) root->color = COLOR_RED;\r\n assignAndFreeIfNull(root, deleteKey(root, key));\r\n if (!isEmpty()) root->color = COLOR_BLACK;\r\n // assert check();\r\n }\r\n\r\n // delete the key-value pair with the given key rooted at h\r\n Node* deleteKey(Node* h, const K& key) {\r\n // assert get(h, key) != null;\r\n if (key < h->key) {\r\n if (!isRed(h->left) && !isRed(h->left->left)) {\r\n h = moveRedLeft(h);\r\n }\r\n assignAndFreeIfNull(h->left, deleteKey(h->left, key));\r\n } else {\r\n if (isRed(h->left)) {\r\n h = rotateRight(h);\r\n }\r\n if (key == h->key && (h->right == nullptr)) {\r\n return nullptr;\r\n }\r\n if (!isRed(h->right) && !isRed(h->right->left)) {\r\n h = moveRedRight(h);\r\n }\r\n if (key == h->key) {\r\n Node* x = min(h->right);\r\n h->key = x->key;\r\n h->val = x->val;\r\n // h->val = get(h->right, min(h->right).key);\r\n // h->key = min(h->right).key;\r\n assignAndFreeIfNull(h->right, deleteMin(h->right));\r\n } else {\r\n assignAndFreeIfNull(h->right, deleteKey(h->right, key));\r\n }\r\n }\r\n return balance(h);\r\n }\r\n\r\n /***************************************************************************\r\n * Red-black tree helper functions.\r\n ***************************************************************************/\r\n\r\n // make a left-leaning link lean to the right\r\n Node* rotateRight(Node* h) {\r\n // assert (h != null) && isRed(h->left);\r\n Node* x = h->left;\r\n h->left = x->right;\r\n x->right = h;\r\n x->color = x->right->color;\r\n x->right->color = COLOR_RED;\r\n x->size = h->size;\r\n h->size = size(h->left) + size(h->right) + 1;\r\n return x;\r\n }\r\n\r\n // make a right-leaning link lean to the left\r\n Node* rotateLeft(Node* h) {\r\n // assert (h != null) && isRed(h->right);\r\n Node* x = h->right;\r\n h->right = x->left;\r\n x->left = h;\r\n x->color = x->left->color;\r\n x->left->color = COLOR_RED;\r\n x->size = h->size;\r\n h->size = size(h->left) + size(h->right) + 1;\r\n return x;\r\n }\r\n\r\n // flip the colors of a node and its two children\r\n void flipColors(Node* h) {\r\n // h must have opposite color of its two children\r\n // assert (h != null) && (h->left != null) && (h->right != null);\r\n // assert (!isRed(h) && isRed(h->left) && isRed(h->right))\r\n // || (isRed(h) && !isRed(h->left) && !isRed(h->right));\r\n h->color = !h->color;\r\n h->left->color = !h->left->color;\r\n h->right->color = !h->right->color;\r\n }\r\n\r\n // Assuming that h is red and both h->left and h->left.left\r\n // are black, make h->left or one of its children red.\r\n Node* moveRedLeft(Node* h) {\r\n // assert (h != null);\r\n // assert isRed(h) && !isRed(h->left) && !isRed(h->left.left);\r\n\r\n flipColors(h);\r\n if (isRed(h->right->left)) {\r\n h->right = rotateRight(h->right);\r\n h = rotateLeft(h);\r\n flipColors(h);\r\n }\r\n return h;\r\n }\r\n\r\n // Assuming that h is red and both h->right and h->right.left\r\n // are black, make h->right or one of its children red.\r\n Node* moveRedRight(Node* h) {\r\n // assert (h != null);\r\n // assert isRed(h) && !isRed(h->right) && !isRed(h->right.left);\r\n flipColors(h);\r\n if (isRed(h->left->left)) {\r\n h = rotateRight(h);\r\n flipColors(h);\r\n }\r\n return h;\r\n }\r\n\r\n // restore red-black tree invariant\r\n Node* balance(Node* h) {\r\n // assert (h != null);\r\n\r\n if (isRed(h->right)) h = rotateLeft(h);\r\n if (isRed(h->left) && isRed(h->left->left)) h = rotateRight(h);\r\n if (isRed(h->left) && isRed(h->right)) flipColors(h);\r\n\r\n h->size = size(h->left) + size(h->right) + 1;\r\n return h;\r\n }\r\n\r\n\r\n /***************************************************************************\r\n * Utility functions.\r\n ***************************************************************************/\r\n\r\n /**\r\n * Returns the height of the BST (for debugging).\r\n * @return the height of the BST (a 1-node tree has height 0)\r\n */\r\n int height() {\r\n return height(root);\r\n }\r\n int height(Node* x) {\r\n if (x == nullptr) return -1;\r\n return 1 + std::max(height(x->left), height(x->right));\r\n }\r\n\r\n /***************************************************************************\r\n * Ordered symbol table methods.\r\n ***************************************************************************/\r\n\r\n /**\r\n * Returns the smallest key in the symbol table.\r\n * @return the smallest key in the symbol table\r\n * @throws NoSuchElementException if the symbol table is empty\r\n */\r\n K* min() {\r\n if (isEmpty()) return nullptr;\r\n return min(root).key;\r\n }\r\n\r\n // the smallest key in subtree rooted at x; null if no such key\r\n Node* min(Node* x) {\r\n // assert x != null;\r\n if (x->left == nullptr) return x;\r\n else return min(x->left);\r\n }\r\n\r\n /**\r\n * Returns the largest key in the symbol table.\r\n * @return the largest key in the symbol table\r\n * @throws NoSuchElementException if the symbol table is empty\r\n */\r\n K* max() {\r\n if (isEmpty()) return nullptr;\r\n return max(root).key;\r\n }\r\n\r\n // the largest key in the subtree rooted at x; null if no such key\r\n Node* max(Node* x) {\r\n // assert x != null;\r\n if (x->right == nullptr) return x;\r\n else return max(x->right);\r\n }\r\n\r\n\r\n /**\r\n * Returns the largest key in the symbol table less than or equal to {@code key}.\r\n * @param key the key\r\n * @return the largest key in the symbol table less than or equal to {@code key}\r\n * @throws NoSuchElementException if there is no such key\r\n * @throws IllegalArgumentException if {@code key} is {@code null}\r\n */\r\n K* floor(const K& key) {\r\n if (key == nullptr) return nullptr;\r\n if (isEmpty()) return nullptr;\r\n Node* x = floor(root, key);\r\n if (x == nullptr) return nullptr;\r\n else return x->key;\r\n }\r\n\r\n // the largest key in the subtree rooted at x less than or equal to the given key\r\n Node* floor(Node* x, const K& key) {\r\n if (x == nullptr) return nullptr;\r\n if (key == x->key) return x;\r\n if (key < x->key) return floor(x->left, key);\r\n Node* t = floor(x->right, key);\r\n if (t != nullptr) return t;\r\n else return x;\r\n }\r\n\r\n /**\r\n * Returns the smallest key in the symbol table greater than or equal to {@code key}.\r\n * @param key the key\r\n * @return the smallest key in the symbol table greater than or equal to {@code key}\r\n * @throws NoSuchElementException if there is no such key\r\n * @throws IllegalArgumentException if {@code key} is {@code null}\r\n */\r\n K* ceiling(const K& key) {\r\n if (key == nullptr) return nullptr;\r\n if (isEmpty()) return nullptr;\r\n Node* x = ceiling(root, key);\r\n if (x == nullptr) return nullptr;\r\n else return x->key;\r\n }\r\n\r\n // the smallest key in the subtree rooted at x greater than or equal to the given key\r\n Node* ceiling(Node* x, const K& key) {\r\n if (x == nullptr) return nullptr;\r\n if (key == x->key) return x;\r\n if (x->key < key) return ceiling(x->right, key);\r\n Node* t = ceiling(x->left, key);\r\n if (t != nullptr) return t;\r\n else return x;\r\n }\r\n\r\n /**\r\n * Return the kth smallest key in the symbol table.\r\n * @param k the order statistic\r\n * @return the {@code k}th smallest key in the symbol table\r\n * @throws IllegalArgumentException unless {@code k} is between 0 and\r\n * n1\r\n */\r\n K* select(int k) {\r\n if (k < 0 || k >= size()) {\r\n return nullptr;\r\n }\r\n Node x = select(root, k);\r\n return x->key;\r\n }\r\n\r\n // the key of rank k in the subtree rooted at x\r\n Node* select(Node* x, int k) {\r\n // assert x != null;\r\n // assert k >= 0 && k < size(x);\r\n int t = size(x->left);\r\n if (t > k) return select(x->left, k);\r\n else if (t < k) return select(x->right, k-t-1);\r\n else return x;\r\n }\r\n\r\n /**\r\n * Return the number of keys in the symbol table strictly less than {@code key}.\r\n * @param key the key\r\n * @return the number of keys in the symbol table strictly less than {@code key}\r\n * @throws IllegalArgumentException if {@code key} is {@code null}\r\n */\r\n int rank(const K& key) {\r\n if (key == nullptr) return -1;\r\n return rank(key, root);\r\n }\r\n\r\n // number of keys less than key in the subtree rooted at x\r\n int rank(const K& key, Node* x) {\r\n if (x == nullptr) return 0;\r\n if (key < x->key) return rank(key, x->left);\r\n else if (x->key < key) return 1 + size(x->left) + rank(key, x->right);\r\n else return size(x->left);\r\n }\r\n\r\n /***************************************************************************\r\n * Range count and range search->\r\n ***************************************************************************/\r\n\r\n\r\n /**\r\n * Returns the number of keys in the symbol table in the given range.\r\n *\r\n * @param lo minimum endpoint\r\n * @param hi maximum endpoint\r\n * @return the number of keys in the sybol table between {@code lo}\r\n * (inclusive) and {@code hi} (inclusive)\r\n * @throws IllegalArgumentException if either {@code lo} or {@code hi}\r\n * is {@code null}\r\n */\r\n int size(const K& lo, const K& hi) {\r\n if (lo == nullptr) return 0;\r\n if (hi == nullptr) return 0;\r\n\r\n if (hi < lo) return 0;\r\n if (containsKey(hi)) return rank(hi) - rank(lo) + 1;\r\n else return rank(hi) - rank(lo);\r\n }\r\n\r\n\r\n /***************************************************************************\r\n * Check integrity of red-black tree data structure.\r\n ***************************************************************************/\r\n bool check() {\r\n if (!isBST()) std::cout << \"Not in symmetric order\\n\";\r\n if (!isSizeConsistent()) std::cout << \"Subtree counts not consistent\\n\";\r\n //if (!isRankConsistent()) std::cout << \"Ranks not consistent\\n\";\r\n if (!is23()) std::cout << \"Not a 2-3 tree\\n\";\r\n if (!isBalanced()) std::cout << \"Not balanced\\n\";\r\n return isBST() && isSizeConsistent() && is23() && isBalanced();\r\n }\r\n\r\n // does this binary tree satisfy symmetric order?\r\n // Note: this test also ensures that data structure is a binary tree since order is strict\r\n bool isBST() {\r\n return isBST(root, nullptr, nullptr);\r\n }\r\n\r\n // is the tree rooted at x a BST with all keys strictly between min and max\r\n // (if min or max is null, treat as empty constraint)\r\n // Credit: Bob Dondero's elegant solution\r\n bool isBST(Node* x, K* min, K* max) {\r\n if (x == nullptr) return true;\r\n // TODO: port these two lines\r\n //if (min != nullptr && x->key.compareTo(min) <= 0) return false;\r\n //if (max != nullptr && x->key.compareTo(max) >= 0) return false;\r\n return isBST(x->left, min, x->key) && isBST(x->right, x->key, max);\r\n }\r\n\r\n // are the size fields correct?\r\n bool isSizeConsistent() { return isSizeConsistent(root); }\r\n bool isSizeConsistent(Node* x) {\r\n if (x == nullptr) return true;\r\n if (x->size != size(x->left) + size(x->right) + 1) return false;\r\n return isSizeConsistent(x->left) && isSizeConsistent(x->right);\r\n }\r\n\r\n /*\r\n // check that ranks are consistent\r\n bool isRankConsistent() {\r\n for (int i = 0; i < size(); i++)\r\n if (i != rank(select(i))) return false;\r\n for (K* key : keys())\r\n if (key.compareTo(select(rank(key))) != 0) return false;\r\n return true;\r\n }\r\n */\r\n\r\n // Does the tree have no red right links, and at most one (left)\r\n // red links in a row on any path?\r\n bool is23() { return is23(root); }\r\n bool is23(Node* x) {\r\n if (x == nullptr) return true;\r\n if (isRed(x->right)) return false;\r\n if (x != root && isRed(x) && isRed(x->left))\r\n return false;\r\n return is23(x->left) && is23(x->right);\r\n }\r\n\r\n // do all paths from root to leaf have same number of black edges?\r\n bool isBalanced() {\r\n int black = 0; // number of black links on path from root to min\r\n Node x = root;\r\n while (x != nullptr) {\r\n if (!isRed(x)) black++;\r\n x = x->left;\r\n }\r\n return isBalanced(root, black);\r\n }\r\n\r\n // does every path from the root to a leaf have the given number of black links?\r\n bool isBalanced(Node* x, int black) {\r\n if (x == nullptr) return black == 0;\r\n if (!isRed(x)) black--;\r\n return isBalanced(x->left, black) && isBalanced(x->right, black);\r\n }\r\n\r\n\r\n\r\n // Inserts a key only if it's not already present\r\n bool add(const K& key, const int tid=0) {\r\n bool retval = false;\r\n estm::updateTx([&] () {\r\n \tretval = innerPut(key,key);\r\n });\r\n return retval;\r\n }\r\n\r\n // Returns true only if the key was present\r\n bool remove(K& key, const int tid=0) {\r\n bool retval = false;\r\n estm::updateTx([&] () {\r\n V notused;\r\n retval = innerGet(key,notused,false);\r\n if (retval) innerRemove(key);\r\n });\r\n return retval;\r\n }\r\n\r\n bool contains(K& key, const int tid=0) {\r\n bool retval = false;\r\n estm::readTx([&] () {\r\n V notused;\r\n retval = innerGet(key,notused,false);\r\n });\r\n return retval;\r\n }\r\n\r\n // This is not fully transactionally but it's ok because we use it only on initialization.\r\n // We could make it fully transactionally, but we would have to increase the size of allocation/store logs.\r\n void addAll(K** keys, int size, const int tid=0) {\r\n for (int i = 0; i < size; i++) add(*keys[i], tid);\r\n }\r\n\r\n static std::string className() { return estm::ESTM::className() + \"-RedBlackTree\"; }\r\n\r\n};\r\n\r\n#endif // _ESTM_RED_BLACK_BST_H_\r\n" }, { "path": "datastructures/treemaps/HazardEras.hpp", "content": "/******************************************************************************\r\n * Copyright (c) 2016-2017, Pedro Ramalhete, Andreia Correia\r\n * All rights reserved.\r\n *\r\n * Redistribution and use in source and binary forms, with or without\r\n * modification, are permitted provided that the following conditions are met:\r\n * * Redistributions of source code must retain the above copyright\r\n * notice, this list of conditions and the following disclaimer.\r\n * * Redistributions in binary form must reproduce the above copyright\r\n * notice, this list of conditions and the following disclaimer in the\r\n * documentation and/or other materials provided with the distribution.\r\n * * Neither the name of Concurrency Freaks nor the\r\n * names of its contributors may be used to endorse or promote products\r\n * derived from this software without specific prior written permission.\r\n\r\n *\r\n * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\r\n * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED\r\n * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\r\n * DISCLAIMED. IN NO EVENT SHALL BE LIABLE FOR ANY\r\n * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\r\n * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\r\n * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND\r\n * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\r\n * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\r\n * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\r\n ******************************************************************************\r\n */\r\n\r\n#ifndef _HAZARD_ERAS_H_\r\n#define _HAZARD_ERAS_H_\r\n\r\n#include \r\n#include \r\n#include \r\n#include \r\n\r\n/*\r\n *

Hazard Eras

\r\n * This a light-weight implementation of hazard eras, where each thread has a\r\n * thread-local list of retired objects.\r\n *\r\n * This is based on the paper \"Hazard Eras - Non-Blocking Memory Reclamation\"\r\n * by Pedro Ramalhete and Andreia Correia:\r\n * https://github.com/pramalhe/ConcurrencyFreaks/blob/master/papers/hazarderas-2017.pdf\r\n *\r\n * The type T is for the objects/nodes and it's it must have the members newEra, delEra\r\n *\r\n * R is zero.\r\n *\r\n *

\r\n * @author Pedro Ramalhete\r\n * @author Andreia Correia\r\n */\r\ntemplate\r\nclass HazardEras {\r\n\r\nprivate:\r\n static const uint64_t NONE = 0;\r\n static const int HE_MAX_THREADS = 128;\r\n static const int MAX_HES = 5; // This is named 'K' in the HP paper\r\n static const int CLPAD = 128/sizeof(std::atomic);\r\n static const int HE_THRESHOLD_R = 0; // This is named 'R' in the HP paper\r\n\r\n const int maxHEs;\r\n const int maxThreads;\r\n\r\n alignas(128) std::atomic eraClock {1};\r\n alignas(128) std::atomic* he[HE_MAX_THREADS];\r\n // It's not nice that we have a lot of empty vectors, but we need padding to avoid false sharing\r\n alignas(128) std::vector retiredList[HE_MAX_THREADS*CLPAD];\r\n\r\npublic:\r\n HazardEras(int maxHEs=MAX_HES, int maxThreads=HE_MAX_THREADS) : maxHEs{maxHEs}, maxThreads{maxThreads} {\r\n for (int it = 0; it < HE_MAX_THREADS; it++) {\r\n he[it] = new std::atomic[CLPAD*2]; // We allocate four cache lines to allow for many hps and without false sharing\r\n retiredList[it*CLPAD].reserve(maxThreads*maxHEs);\r\n for (int ihe = 0; ihe < MAX_HES; ihe++) {\r\n he[it][ihe].store(NONE, std::memory_order_relaxed);\r\n }\r\n }\r\n static_assert(std::is_same::value, \"T::newEra must be uint64_t\");\r\n static_assert(std::is_same::value, \"T::delEra must be uint64_t\");\r\n }\r\n\r\n ~HazardEras() {\r\n for (int it = 0; it < HE_MAX_THREADS; it++) {\r\n delete[] he[it];\r\n // Clear the current retired nodes\r\n for (unsigned iret = 0; iret < retiredList[it*CLPAD].size(); iret++) {\r\n delete retiredList[it*CLPAD][iret];\r\n }\r\n }\r\n }\r\n\r\n\r\n inline uint64_t getEra() {\r\n return eraClock.load();\r\n }\r\n\r\n\r\n /**\r\n * Progress Condition: wait-free bounded (by maxHEs)\r\n */\r\n inline void clear(const int tid) {\r\n for (int ihe = 0; ihe < maxHEs; ihe++) {\r\n he[tid][ihe].store(NONE, std::memory_order_release);\r\n }\r\n }\r\n\r\n\r\n /**\r\n * Progress Condition: lock-free\r\n */\r\n inline T* get_protected(int index, const std::atomic& atom, const int tid) {\r\n auto prevEra = he[tid][index].load(std::memory_order_relaxed);\r\n\t\twhile (true) {\r\n\t\t T* ptr = atom.load();\r\n\t\t auto era = eraClock.load(std::memory_order_acquire);\r\n\t\t if (era == prevEra) return ptr;\r\n he[tid][index].store(era);\r\n prevEra = era;\r\n\t\t}\r\n }\r\n\r\n inline void protectEraRelease(int index, int other, const int tid) {\r\n auto era = he[tid][other].load(std::memory_order_relaxed);\r\n if (he[tid][index].load(std::memory_order_relaxed) == era) return;\r\n he[tid][index].store(era, std::memory_order_release);\r\n }\r\n\r\n\r\n /*\r\n * Does a single iteration. Must be integrated into the algorithm that's using HE.\r\n * In other words, we must re-check if era has changed\r\n *\r\n * Progress Condition: wait-free population oblivious\r\n */\r\n inline T* protectPtr(int index, const std::atomic& atom, uint64_t& prevEra, const int tid) {\r\n T* ptr = atom.load(std::memory_order_acquire);\r\n auto era = eraClock.load();\r\n if (prevEra != era) {\r\n prevEra = era;\r\n he[tid][index].store(era, std::memory_order_relaxed);\r\n std::atomic_thread_fence(std::memory_order_seq_cst);\r\n }\r\n return ptr;\r\n }\r\n\r\n\r\n /**\r\n * Retire an object (node)\r\n * Progress Condition: wait-free bounded\r\n *\r\n * Doing rlist.erase() is not the most efficient way to remove entries from a std::vector, but ok...\r\n */\r\n void retire(T* ptr, const int mytid) {\r\n auto currEra = eraClock.load();\r\n ptr->delEra = currEra;\r\n auto& rlist = retiredList[mytid*CLPAD];\r\n rlist.push_back(ptr);\r\n if (eraClock == currEra) eraClock.fetch_add(1);\r\n for (unsigned iret = 0; iret < rlist.size();) {\r\n auto obj = rlist[iret];\r\n if (canDelete(obj, mytid)) {\r\n rlist.erase(rlist.begin() + iret);\r\n delete obj;\r\n continue;\r\n }\r\n iret++;\r\n }\r\n }\r\n\r\nprivate:\r\n bool canDelete(T* obj, const int mytid) {\r\n for (int tid = 0; tid < maxThreads; tid++) {\r\n for (int ihe = 0; ihe < maxHEs; ihe++) {\r\n const auto era = he[tid][ihe].load(std::memory_order_acquire);\r\n if (era == NONE || era < obj->newEra || era > obj->delEra) continue;\r\n return false;\r\n }\r\n }\r\n return true;\r\n }\r\n\r\n};\r\n\r\n#endif /* _HAZARD_ERAS_H_ */\r\n" }, { "path": "datastructures/treemaps/NatarajanTreeHE.hpp", "content": "/*\r\n\r\nCopyright 2017 University of Rochester\r\n\r\nLicensed under the Apache License, Version 2.0 (the \"License\");\r\nyou may not use this file except in compliance with the License.\r\nYou may obtain a copy of the License at\r\n\r\nhttp://www.apache.org/licenses/LICENSE-2.0\r\n\r\nUnless required by applicable law or agreed to in writing, software\r\ndistributed under the License is distributed on an \"AS IS\" BASIS,\r\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\r\nSee the License for the specific language governing permissions and\r\nlimitations under the License.\r\n\r\nAdapted from https://github.com/roghnin/Interval-Based-Reclamation/blob/master/src/rideables/NatarajanTree.hpp\r\n\r\nDue to the usage of , this needs C++17 to compile\r\n\r\nPedro: I've adapted this for our benchmarks but the adaptation may contain errors, please do not use this code in production!\r\n*/\r\n\r\n\r\n#ifndef _NATARAJAN_TREE_HAZARD_ERAS_H_\r\n#define _NATARAJAN_TREE_HAZARD_ERAS_H_\r\n\r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \"common/HazardEras.hpp\"\r\n\r\n\r\ntemplate \r\nclass NatarajanTreeHE {\r\nprivate:\r\n const int MAX_THREADS = 128;\r\n /* structs*/\r\n struct Node {\r\n int level;\r\n K key;\r\n V val;\r\n std::atomic left;\r\n std::atomic right;\r\n uint64_t newEra {0}; // TODO: put he.getEra() here\r\n uint64_t delEra;\r\n Node(uint64_t newEra) : newEra{newEra} {};\r\n Node(uint64_t newEra, K k, V v, Node* l, Node* r,int lev):level(lev),key(k),val(v),left(l),right(r),newEra{newEra} {};\r\n Node(uint64_t newEra, K k, V v, Node* l, Node* r):level(-1),key(k),val(v),left(l),right(r),newEra{newEra} {};\r\n };\r\n struct SeekRecord{\r\n Node* ancestor;\r\n Node* successor;\r\n Node* parent;\r\n Node* leaf;\r\n };\r\n\r\n /* variables */\r\n HazardEras he {5, MAX_THREADS};\r\n\r\n K infK{};\r\n V defltV{};\r\n Node* r;\r\n Node* s;\r\n SeekRecord* records;\r\n const size_t GET_POINTER_BITS = 0xfffffffffffffffc;//for machine 64-bit or less.\r\n\r\n /* helper functions */\r\n //flag and tags helpers\r\n inline Node* getPtr(Node* mptr){\r\n return (Node*) ((size_t)mptr & GET_POINTER_BITS);\r\n }\r\n inline bool getFlg(Node* mptr){\r\n return (bool)((size_t)mptr & 1);\r\n }\r\n inline bool getTg(Node* mptr){\r\n return (bool)((size_t)mptr & 2);\r\n }\r\n inline Node* mixPtrFlgTg(Node* ptr, bool flg, bool tg){\r\n return (Node*) ((size_t)ptr | flg | ((size_t)tg<<1));\r\n }\r\n //node comparison\r\n inline bool isInf(Node* n){\r\n return getInfLevel(n)!=-1;\r\n }\r\n inline int getInfLevel(Node* n){\r\n //0 for inf0, 1 for inf1, 2 for inf2, -1 for general val\r\n n=getPtr(n);\r\n return n->level;\r\n }\r\n inline bool nodeLess(Node* n1, Node* n2){\r\n n1=getPtr(n1);\r\n n2=getPtr(n2);\r\n int i1=getInfLevel(n1);\r\n int i2=getInfLevel(n2);\r\n return i1keykey);\r\n }\r\n inline bool nodeEqual(Node* n1, Node* n2){\r\n n1=getPtr(n1);\r\n n2=getPtr(n2);\r\n int i1=getInfLevel(n1);\r\n int i2=getInfLevel(n2);\r\n if(i1==-1&&i2==-1)\r\n return n1->key==n2->key;\r\n else\r\n return i1==i2;\r\n }\r\n inline bool nodeLessEqual(Node* n1, Node* n2){\r\n return !nodeLess(n2,n1);\r\n }\r\n\r\n /* private interfaces */\r\n void seek(K key, int tid);\r\n bool cleanup(K key, int tid);\r\n void doRangeQuery(Node& k1, Node& k2, int tid, Node* root, std::map& res);\r\npublic:\r\n NatarajanTreeHE(const int maxThreads=0) {\r\n r = new Node(he.getEra(), infK,defltV,nullptr,nullptr,2);\r\n s = new Node(he.getEra(), infK,defltV,nullptr,nullptr,1);\r\n r->right = new Node(he.getEra(), infK,defltV,nullptr,nullptr,2);\r\n r->left = s;\r\n s->right = new Node(he.getEra(), infK,defltV,nullptr,nullptr,1);\r\n s->left = new Node(he.getEra(), infK,defltV,nullptr,nullptr,0);\r\n records = new SeekRecord[MAX_THREADS]{};\r\n };\r\n ~NatarajanTreeHE(){};\r\n\r\n static std::string className() { return \"NatarajanTreeHE\"; }\r\n std::optional get(K key, int tid);\r\n std::optional put(K key, V val, int tid);\r\n bool insert(K key, V val, int tid);\r\n std::optional innerRemove(K key, int tid);\r\n std::optional replace(K key, V val, int tid);\r\n std::map rangeQuery(K key1, K key2, int& len, int tid);\r\n\r\n // Used only by our tree benchmarks\r\n bool add(K key, int tid);\r\n bool remove(K key, int tid);\r\n bool contains(K key, int tid);\r\n void addAll(K** keys, const int size, const int tid);\r\n};\r\n\r\n//-------Definition----------\r\ntemplate \r\nvoid NatarajanTreeHE::seek(K key, int tid){\r\n /* initialize the seek record using sentinel nodes */\r\n Node keyNode{he.getEra(),key,defltV,nullptr,nullptr};//node to be compared\r\n SeekRecord* seekRecord = &(records[tid]);\r\n seekRecord->ancestor = r;\r\n seekRecord->successor = he.get_protected(1, r->left, tid);\r\n seekRecord->parent = he.get_protected(2, r->left, tid);\r\n seekRecord->leaf = getPtr(he.get_protected(3, s->left, tid));\r\n\r\n /* initialize other variables used in the traversal */\r\n Node* parentField = he.get_protected(3, seekRecord->parent->left, tid);\r\n Node* currentField = he.get_protected(4, seekRecord->leaf->left,tid);\r\n Node* current = getPtr(currentField);\r\n\r\n /* traverse the tree */\r\n while(current!=nullptr){\r\n /* check if the edge from the current parent node is tagged */\r\n if(!getTg(parentField)){\r\n /*\r\n * found an untagged edge in the access path;\r\n * advance ancestor and successor pointers.\r\n */\r\n seekRecord->ancestor=seekRecord->parent;\r\n he.protectEraRelease(0, 1, tid);\r\n seekRecord->successor=seekRecord->leaf;\r\n he.protectEraRelease(1, 3, tid);\r\n }\r\n\r\n /* advance parent and leaf pointers */\r\n seekRecord->parent = seekRecord->leaf;\r\n he.protectEraRelease(2, 3, tid);\r\n seekRecord->leaf = current;\r\n he.protectEraRelease(3, 4, tid);\r\n\r\n /* update other variables used in traversal */\r\n parentField=currentField;\r\n if(nodeLess(&keyNode,current)){\r\n currentField = he.get_protected(4, current->left, tid);\r\n }\r\n else{\r\n currentField = he.get_protected(4, current->right, tid);\r\n }\r\n current=getPtr(currentField);\r\n }\r\n /* traversal complete */\r\n return;\r\n}\r\n\r\ntemplate \r\nbool NatarajanTreeHE::cleanup(K key, int tid){\r\n Node keyNode{he.getEra(),key,defltV,nullptr,nullptr};//node to be compared\r\n bool res=false;\r\n\r\n /* retrieve addresses stored in seek record */\r\n SeekRecord* seekRecord=&(records[tid]);\r\n Node* ancestor=getPtr(seekRecord->ancestor);\r\n Node* successor=getPtr(seekRecord->successor);\r\n Node* parent=getPtr(seekRecord->parent);\r\n Node* leaf=getPtr(seekRecord->leaf);\r\n\r\n std::atomic* successorAddr=nullptr;\r\n std::atomic* childAddr=nullptr;\r\n std::atomic* siblingAddr=nullptr;\r\n\r\n /* obtain address of field of ancestor node that will be modified */\r\n if(nodeLess(&keyNode,ancestor))\r\n successorAddr=&(ancestor->left);\r\n else\r\n successorAddr=&(ancestor->right);\r\n\r\n /* obtain addresses of child fields of parent node */\r\n if(nodeLess(&keyNode,parent)){\r\n childAddr=&(parent->left);\r\n siblingAddr=&(parent->right);\r\n }\r\n else{\r\n childAddr=&(parent->right);\r\n siblingAddr=&(parent->left);\r\n }\r\n Node* tmpChild=childAddr->load(std::memory_order_acquire);\r\n if(!getFlg(tmpChild)){\r\n /* the leaf is not flagged, thus sibling node should be flagged */\r\n tmpChild=siblingAddr->load(std::memory_order_acquire);\r\n /* switch the sibling address */\r\n siblingAddr=childAddr;\r\n }\r\n\r\n /* use TAS to tag sibling edge */\r\n while(true){\r\n Node* untagged=siblingAddr->load(std::memory_order_acquire);\r\n Node* tagged=mixPtrFlgTg(getPtr(untagged),getFlg(untagged),true);\r\n if(siblingAddr->compare_exchange_strong(untagged,tagged,std::memory_order_acq_rel)){\r\n break;\r\n }\r\n }\r\n /* read the flag and address fields */\r\n Node* tmpSibling=siblingAddr->load(std::memory_order_acquire);\r\n\r\n /* make the sibling node a direct child of the ancestor node */\r\n res=successorAddr->compare_exchange_strong(successor,\r\n mixPtrFlgTg(getPtr(tmpSibling),getFlg(tmpSibling),false),\r\n std::memory_order_acq_rel);\r\n\r\n if(res==true){\r\n he.retire(getPtr(tmpChild),tid);\r\n he.retire(successor,tid);\r\n }\r\n return res;\r\n}\r\n\r\n/* to test rangeQuery */\r\n// template <>\r\n// optional NatarajanTree::get(int key, int tid){\r\n// int len=0;\r\n// auto x = rangeQuery(key-500,key,len,tid);\r\n// Node keyNode{key,defltV,nullptr,nullptr};//node to be compared\r\n// optional res={};\r\n// SeekRecord* seekRecord=&(records[tid].ui);\r\n// Node* leaf=nullptr;\r\n// seek(key,tid);\r\n// leaf=getPtr(seekRecord->leaf);\r\n// if(nodeEqual(&keyNode,leaf)){\r\n// res = leaf->val;\r\n// }\r\n// return res;\r\n// }\r\n\r\ntemplate \r\nstd::optional NatarajanTreeHE::get(K key, int tid){\r\n Node keyNode{he.getEra(),key,defltV,nullptr,nullptr};//node to be compared\r\n std::optional res={};\r\n SeekRecord* seekRecord=&(records[tid]);\r\n Node* leaf=nullptr;\r\n seek(key,tid);\r\n leaf=getPtr(seekRecord->leaf);\r\n if(nodeEqual(&keyNode,leaf)){\r\n res = leaf->val;\r\n }\r\n he.clear(tid);\r\n return res;\r\n}\r\n\r\ntemplate \r\nstd::optional NatarajanTreeHE::put(K key, V val, int tid){\r\n std::optional res={};\r\n SeekRecord* seekRecord=&(records[tid]);\r\n\r\n Node* newInternal=nullptr;\r\n Node* newLeaf = new Node(he.getEra(),key,val,nullptr,nullptr);//also to compare keys\r\n\r\n Node* parent=nullptr;\r\n Node* leaf=nullptr;\r\n std::atomic* childAddr=nullptr;\r\n\r\n while(true){\r\n seek(key,tid);\r\n leaf=getPtr(seekRecord->leaf);\r\n parent=getPtr(seekRecord->parent);\r\n if(!nodeEqual(newLeaf,leaf)){//key does not exist\r\n /* obtain address of the child field to be modified */\r\n if(nodeLess(newLeaf,parent))\r\n childAddr=&(parent->left);\r\n else\r\n childAddr=&(parent->right);\r\n\r\n /* create left and right leave of newInternal */\r\n Node* newLeft=nullptr;\r\n Node* newRight=nullptr;\r\n if(nodeLess(newLeaf,leaf)){\r\n newLeft=newLeaf;\r\n newRight=leaf;\r\n }\r\n else{\r\n newLeft=leaf;\r\n newRight=newLeaf;\r\n }\r\n\r\n /* create newInternal */\r\n if(isInf(leaf)){\r\n int lev=getInfLevel(leaf);\r\n newInternal = new Node(he.getEra(),infK,defltV,newLeft,newRight,lev);\r\n }\r\n else\r\n newInternal = new Node(he.getEra(),std::max(key,leaf->key),defltV,newLeft,newRight);\r\n\r\n /* try to add the new nodes to the tree */\r\n Node* tmpExpected=getPtr(leaf);\r\n if(childAddr->compare_exchange_strong(tmpExpected,getPtr(newInternal),std::memory_order_acq_rel)){\r\n res={};\r\n break;//insertion succeeds\r\n }\r\n else{//fails; help conflicting delete operation\r\n delete newInternal;\r\n Node* tmpChild=childAddr->load(std::memory_order_acquire);\r\n if(getPtr(tmpChild)==leaf && (getFlg(tmpChild)||getTg(tmpChild))){\r\n /*\r\n * address of the child has not changed\r\n * and either the leaf node or its sibling\r\n * has been flagged for deletion\r\n */\r\n cleanup(key,tid);\r\n }\r\n }\r\n }\r\n else{//key exists, update and return old\r\n res=leaf->val;\r\n if(nodeLess(newLeaf,parent))\r\n childAddr=&(parent->left);\r\n else\r\n childAddr=&(parent->right);\r\n if(childAddr->compare_exchange_strong(leaf,newLeaf,std::memory_order_acq_rel)){\r\n he.retire(leaf,tid);\r\n break;\r\n }\r\n }\r\n }\r\n he.clear(tid);\r\n return res;\r\n}\r\n\r\ntemplate \r\nbool NatarajanTreeHE::insert(K key, V val, int tid) {\r\n bool res=false;\r\n SeekRecord* seekRecord=&(records[tid]);\r\n\r\n Node* newInternal=nullptr;\r\n Node* newLeaf = new Node(he.getEra(),key,val,nullptr,nullptr);//also for comparing keys\r\n\r\n Node* parent=nullptr;\r\n Node* leaf=nullptr;\r\n std::atomic* childAddr=nullptr;\r\n while(true){\r\n seek(key,tid);\r\n leaf=getPtr(seekRecord->leaf);\r\n parent=getPtr(seekRecord->parent);\r\n if(!nodeEqual(newLeaf,leaf)){//key does not exist\r\n /* obtain address of the child field to be modified */\r\n if(nodeLess(newLeaf,parent))\r\n childAddr=&(parent->left);\r\n else\r\n childAddr=&(parent->right);\r\n\r\n /* create left and right leave of newInternal */\r\n Node* newLeft=nullptr;\r\n Node* newRight=nullptr;\r\n if(nodeLess(newLeaf,leaf)){\r\n newLeft=newLeaf;\r\n newRight=leaf;\r\n }\r\n else{\r\n newLeft=leaf;\r\n newRight=newLeaf;\r\n }\r\n\r\n /* create newInternal */\r\n if(isInf(leaf)){\r\n int lev=getInfLevel(leaf);\r\n newInternal = new Node(he.getEra(),infK,defltV,newLeft,newRight,lev);\r\n }\r\n else\r\n newInternal = new Node(he.getEra(),std::max(key,leaf->key),defltV,newLeft,newRight);\r\n\r\n /* try to add the new nodes to the tree */\r\n Node* tmpExpected=getPtr(leaf);\r\n if(childAddr->compare_exchange_strong(tmpExpected,getPtr(newInternal),std::memory_order_acq_rel)){\r\n res=true;\r\n break;//insertion succeeds\r\n }\r\n else{//fails; help conflicting delete operation\r\n delete newInternal;\r\n Node* tmpChild=childAddr->load(std::memory_order_acquire);\r\n if(getPtr(tmpChild)==leaf && (getFlg(tmpChild)||getTg(tmpChild))){\r\n /*\r\n * address of the child has not changed\r\n * and either the leaf node or its sibling\r\n * has been flagged for deletion\r\n */\r\n cleanup(key,tid);\r\n }\r\n }\r\n }\r\n else{//key exists, insertion fails\r\n delete newLeaf;\r\n res=false;\r\n break;\r\n }\r\n }\r\n he.clear(tid);\r\n return res;\r\n}\r\n\r\ntemplate \r\nstd::optional NatarajanTreeHE::innerRemove(K key, int tid){\r\n bool injecting = true;\r\n std::optional res={};\r\n SeekRecord* seekRecord=&(records[tid]);\r\n\r\n Node keyNode{he.getEra(),key,defltV,nullptr,nullptr};//node to be compared\r\n\r\n Node* parent=nullptr;\r\n Node* leaf=nullptr;\r\n std::atomic* childAddr=nullptr;\r\n while(true){\r\n seek(key,tid);\r\n parent=getPtr(seekRecord->parent);\r\n /* obtain address of the child field to be modified */\r\n if(nodeLess(&keyNode,parent))\r\n childAddr=&(parent->left);\r\n else\r\n childAddr=&(parent->right);\r\n\r\n if(injecting){\r\n /* injection mode: check if the key exists */\r\n leaf=getPtr(seekRecord->leaf);\r\n if(!nodeEqual(leaf,&keyNode)){//does not exist\r\n res={};\r\n break;\r\n }\r\n\r\n /* inject the delete operation into the tree */\r\n Node* tmpExpected=getPtr(leaf);\r\n res=leaf->val;\r\n if(childAddr->compare_exchange_strong(tmpExpected,\r\n mixPtrFlgTg(tmpExpected,true,false), std::memory_order_acq_rel)){\r\n /* advance to cleanup mode to remove the leaf node */\r\n injecting=false;\r\n if(cleanup(key,tid)) break;\r\n }\r\n else{\r\n Node* tmpChild=childAddr->load(std::memory_order_acquire);\r\n if(getPtr(tmpChild)==leaf && (getFlg(tmpChild)||getTg(tmpChild))){\r\n /*\r\n * address of the child has not\r\n * changed and either the leaf\r\n * node or its sibling has been\r\n * flagged for deletion\r\n */\r\n cleanup(key,tid);\r\n }\r\n }\r\n }\r\n else{\r\n /* cleanup mode: check if flagged node still exists */\r\n if(seekRecord->leaf!=leaf){\r\n /* leaf no longer in the tree */\r\n break;\r\n }\r\n else{\r\n /* leaf still in the tree; remove */\r\n if(cleanup(key,tid)) break;\r\n }\r\n }\r\n }\r\n he.clear(tid);\r\n return res;\r\n}\r\n\r\ntemplate \r\nstd::optional NatarajanTreeHE::replace(K key, V val, int tid){\r\n std::optional res={};\r\n SeekRecord* seekRecord=&(records[tid]);\r\n\r\n Node* newInternal=nullptr;\r\n Node* newLeaf = new Node(he.getEra(),key,val,nullptr,nullptr);//also to compare keys\r\n\r\n Node* parent=nullptr;\r\n Node* leaf=nullptr;\r\n std::atomic* childAddr=nullptr;\r\n while(true){\r\n seek(key,tid);\r\n parent=getPtr(seekRecord->parent);\r\n leaf=getPtr(seekRecord->leaf);\r\n if(!nodeEqual(newLeaf,leaf)){//key does not exist, replace fails\r\n delete newLeaf;\r\n res={};\r\n break;\r\n }\r\n else{//key exists, update and return old\r\n res=leaf->val;\r\n if(nodeLess(newLeaf,parent))\r\n childAddr=&(parent->left);\r\n else\r\n childAddr=&(parent->right);\r\n if(childAddr->compare_exchange_strong(leaf,newLeaf,std::memory_order_acq_rel)){\r\n he.retire(leaf,tid);\r\n break;\r\n }\r\n }\r\n }\r\n he.clear(tid);\r\n return res;\r\n}\r\n\r\ntemplate \r\nstd::map NatarajanTreeHE::rangeQuery(K key1, K key2, int& len, int tid){\r\n //NOT HP-like GC safe.\r\n if(key1>key2) return {};\r\n Node k1{he.getEra(),key1,defltV,nullptr,nullptr};//node to be compared\r\n Node k2{he.getEra(),key2,defltV,nullptr,nullptr};//node to be compared\r\n\r\n Node* leaf = getPtr(he.get_protected(0, s->left, tid));\r\n Node* current = getPtr(he.get_protected(1, leaf->left, tid));\r\n\r\n std::map res;\r\n if(current!=nullptr)\r\n doRangeQuery(k1,k2,tid,current,res);\r\n len=res.size();\r\n return res;\r\n}\r\n\r\ntemplate \r\nvoid NatarajanTreeHE::doRangeQuery(Node& k1, Node& k2, int tid, Node* root, std::map& res){\r\n Node* left = getPtr(he.get_protected(2, root->left, tid));\r\n Node* right = getPtr(he.get_protected(3, root->right, tid));\r\n if(left==nullptr&&right==nullptr){\r\n if(nodeLessEqual(&k1,root)&&nodeLessEqual(root,&k2)){\r\n res.emplace(root->key,root->val);\r\n }\r\n return;\r\n }\r\n if(left!=nullptr){\r\n if(nodeLess(&k1,root)){\r\n doRangeQuery(k1,k2,tid,left,res);\r\n }\r\n }\r\n if(right!=nullptr){\r\n if(nodeLessEqual(root,&k2)){\r\n doRangeQuery(k1,k2,tid,right,res);\r\n }\r\n }\r\n return;\r\n}\r\n\r\n\r\n// Wrappers for the \"set\" benchmarks\r\ntemplate \r\nbool NatarajanTreeHE::add(K key, int tid) {\r\n return insert(key,key,tid);\r\n}\r\n\r\ntemplate \r\nbool NatarajanTreeHE::remove(K key, int tid) {\r\n return innerRemove(key,tid).has_value();\r\n}\r\n\r\ntemplate \r\nbool NatarajanTreeHE::contains(K key, int tid) {\r\n return get(key,tid).has_value();\r\n}\r\n\r\n// Not lock-free\r\ntemplate \r\nvoid NatarajanTreeHE::addAll(K** keys, const int size, const int tid) {\r\n for (int i = 0; i < size; i++) add(*keys[i], tid);\r\n}\r\n\r\n#endif\r\n" }, { "path": "datastructures/treemaps/OFLFRedBlackTree.hpp", "content": "#ifndef _OF_LF_RED_BLACK_BST_H_\r\n#define _OF_LF_RED_BLACK_BST_H_\r\n\r\n#include \r\n#include \r\n#include \r\n\r\n#include \"stms/OneFileLF.hpp\" // This header defines the macros for the STM being compiled\r\n\r\n// Adapted from Java to C++ from the original at http://algs4.cs.princeton.edu/code/edu/princeton/cs/algs4/RedBlackBST.java\r\ntemplate\r\nclass OFLFRedBlackTree {\r\n const int64_t COLOR_RED = 0;\r\n const int64_t COLOR_BLACK = 1;\r\n\r\n struct Node : public oflf::tmbase {\r\n oflf::tmtype key;\r\n oflf::tmtype val;\r\n oflf::tmtype left {nullptr};\r\n oflf::tmtype right {nullptr};\r\n oflf::tmtype color; // color of parent link\r\n oflf::tmtype size; // subtree count\r\n Node(const K& key, const V& val, int64_t color, int64_t size) : key{key}, val{val}, color{color}, size{size} {}\r\n };\r\n\r\n oflf::tmtype root {nullptr}; // root of the BST\r\n\r\n inline void assignAndFreeIfNull(oflf::tmtype& z, Node* w) {\r\n Node* tofree = z;\r\n z = w;\r\n if (w == nullptr) oflf::tmDelete(tofree);\r\n }\r\n\r\npublic:\r\n /**\r\n * Initializes an empty symbol table.\r\n */\r\n OFLFRedBlackTree(int numThreads=0){ }\r\n\r\n ~OFLFRedBlackTree() {\r\n for (int i = 0; i < 10000; i++) {\r\n oflf::updateTx([&] () {\r\n if (root == nullptr) return;\r\n deleteMin();\r\n });\r\n }\r\n }\r\n\r\n /***************************************************************************\r\n * Node helper methods.\r\n ***************************************************************************/\r\n // is node x red; false if x is null ?\r\n bool isRed(Node* x) {\r\n if (x == nullptr) return false;\r\n return x->color == COLOR_RED;\r\n }\r\n\r\n // number of node in subtree rooted at x; 0 if x is null\r\n int size(Node* x) {\r\n if (x == nullptr) return 0;\r\n return x->size;\r\n }\r\n\r\n\r\n /**\r\n * Returns the number of key-value pairs in this symbol table.\r\n * @return the number of key-value pairs in this symbol table\r\n */\r\n int size() {\r\n return size(root);\r\n }\r\n\r\n /**\r\n * Is this symbol table empty?\r\n * @return {@code true} if this symbol table is empty and {@code false} otherwise\r\n */\r\n bool isEmpty() {\r\n return root == nullptr;\r\n }\r\n\r\n\r\n /***************************************************************************\r\n * Standard BST search->\r\n ***************************************************************************/\r\n\r\n /**\r\n * Returns the value associated with the given key.\r\n * @param key the key\r\n * @return the value associated with the given key if the key is in the symbol table\r\n * and {@code null} if the key is not in the symbol table\r\n * @throws IllegalArgumentException if {@code key} is {@code null}\r\n */\r\n bool innerGet(K& key, V& oldValue, const bool saveOldValue) {\r\n bool found = get(root, key);\r\n if (!found) return false;\r\n //if (saveOldValue) oldValue = *val; // Copy of V\r\n return true;\r\n }\r\n\r\n // value associated with the given key in subtree rooted at x; null if no such key\r\n bool get(Node* x, K& key) {\r\n while (x != nullptr) {\r\n if (key < x->key) x = x->left;\r\n else if (x->key < key) x = x->right;\r\n else return true;\r\n }\r\n return false;\r\n }\r\n\r\n /**\r\n * Does this symbol table contain the given key?\r\n * @param key the key\r\n * @return {@code true} if this symbol table contains {@code key} and\r\n * {@code false} otherwise\r\n * @throws IllegalArgumentException if {@code key} is {@code null}\r\n */\r\n bool containsKey(const K& key) {\r\n return get(key) != nullptr;\r\n }\r\n\r\n /***************************************************************************\r\n * Red-black tree insertion.\r\n ***************************************************************************/\r\n\r\n /**\r\n * Inserts the specified key-value pair into the symbol table, overwriting the old\r\n * value with the new value if the symbol table already contains the specified key.\r\n * Deletes the specified key (and its associated value) from this symbol table\r\n * if the specified value is {@code null}.\r\n *\r\n * @param key the key\r\n * @param val the value\r\n * @throws IllegalArgumentException if {@code key} is {@code null}\r\n */\r\n bool innerPut(const K& key, const V& value) {\r\n \tbool ret = false;\r\n root = put(root, key, value, ret);\r\n root->color = COLOR_BLACK;\r\n return ret;\r\n }\r\n\r\n // insert the key-value pair in the subtree rooted at h\r\n Node* put(Node* h, const K& key, const V& val, bool& ret) {\r\n if (h == nullptr) {\r\n ret = true;\r\n return oflf::tmNew(key, val, COLOR_RED, 1);\r\n }\r\n if (key < h->key) h->left = put(h->left, key, val, ret);\r\n else if (h->key < key) h->right = put(h->right, key, val, ret);\r\n else h->val = val;\r\n // fix-up any right-leaning links\r\n if (isRed(h->right) && !isRed(h->left)) h = rotateLeft(h);\r\n if (isRed(h->left) && isRed(h->left->left)) h = rotateRight(h);\r\n if (isRed(h->left) && isRed(h->right)) flipColors(h);\r\n h->size = size(h->left) + size(h->right) + 1;\r\n\r\n return h;\r\n }\r\n\r\n /***************************************************************************\r\n * Red-black tree deletion.\r\n ***************************************************************************/\r\n\r\n /**\r\n * Removes the smallest key and associated value from the symbol table.\r\n * @throws NoSuchElementException if the symbol table is empty\r\n */\r\n void deleteMin() {\r\n if (isEmpty()) return;\r\n // if both children of root are black, set root to red\r\n if (!isRed(root->left) && !isRed(root->right))\r\n root->color = COLOR_RED;\r\n assignAndFreeIfNull(root, deleteMin(root));\r\n if (!isEmpty()) root->color = COLOR_BLACK;\r\n // assert check();\r\n }\r\n\r\n // delete the key-value pair with the minimum key rooted at h\r\n Node* deleteMin(Node* h) {\r\n if (h->left == nullptr)\r\n return nullptr;\r\n if (!isRed(h->left) && !isRed(h->left->left))\r\n h = moveRedLeft(h);\r\n assignAndFreeIfNull(h->left, deleteMin(h->left));\r\n return balance(h);\r\n }\r\n\r\n\r\n /**\r\n * Removes the largest key and associated value from the symbol table.\r\n * @throws NoSuchElementException if the symbol table is empty\r\n */\r\n void deleteMax() {\r\n if (isEmpty()) return;\r\n\r\n // if both children of root are black, set root to red\r\n if (!isRed(root->left) && !isRed(root->right))\r\n root->color = COLOR_RED;\r\n\r\n root = deleteMax(root);\r\n if (!isEmpty()) root->color = COLOR_BLACK;\r\n // assert check();\r\n }\r\n\r\n // delete the key-value pair with the maximum key rooted at h\r\n Node* deleteMax(Node* h) {\r\n if (isRed(h->left))\r\n h = rotateRight(h);\r\n\r\n if (h->right == nullptr)\r\n return nullptr;\r\n\r\n if (!isRed(h->right) && !isRed(h->right->left))\r\n h = moveRedRight(h);\r\n\r\n h->right = deleteMax(h->right);\r\n\r\n return balance(h);\r\n }\r\n\r\n /**\r\n * Removes the specified key and its associated value from this symbol table\r\n * (if the key is in this symbol table).\r\n *\r\n * @param key the key\r\n */\r\n void innerRemove(const K& key) {\r\n // if both children of root are black, set root to red\r\n if (!isRed(root->left) && !isRed(root->right)) root->color = COLOR_RED;\r\n assignAndFreeIfNull(root, deleteKey(root, key));\r\n if (!isEmpty()) root->color = COLOR_BLACK;\r\n // assert check();\r\n }\r\n\r\n // delete the key-value pair with the given key rooted at h\r\n Node* deleteKey(Node* h, const K& key) {\r\n // assert get(h, key) != null;\r\n if (key < h->key) {\r\n if (!isRed(h->left) && !isRed(h->left->left)) {\r\n h = moveRedLeft(h);\r\n }\r\n assignAndFreeIfNull(h->left, deleteKey(h->left, key));\r\n } else {\r\n if (isRed(h->left)) {\r\n h = rotateRight(h);\r\n }\r\n if (key == h->key && (h->right == nullptr)) {\r\n return nullptr;\r\n }\r\n if (!isRed(h->right) && !isRed(h->right->left)) {\r\n h = moveRedRight(h);\r\n }\r\n if (key == h->key) {\r\n Node* x = min(h->right);\r\n h->key = x->key;\r\n h->val = x->val;\r\n // h->val = get(h->right, min(h->right).key);\r\n // h->key = min(h->right).key;\r\n assignAndFreeIfNull(h->right, deleteMin(h->right));\r\n } else {\r\n assignAndFreeIfNull(h->right, deleteKey(h->right, key));\r\n }\r\n }\r\n return balance(h);\r\n }\r\n\r\n /***************************************************************************\r\n * Red-black tree helper functions.\r\n ***************************************************************************/\r\n\r\n // make a left-leaning link lean to the right\r\n Node* rotateRight(Node* h) {\r\n // assert (h != null) && isRed(h->left);\r\n Node* x = h->left;\r\n h->left = x->right;\r\n x->right = h;\r\n x->color = x->right->color;\r\n x->right->color = COLOR_RED;\r\n x->size = h->size;\r\n h->size = size(h->left) + size(h->right) + 1;\r\n return x;\r\n }\r\n\r\n // make a right-leaning link lean to the left\r\n Node* rotateLeft(Node* h) {\r\n // assert (h != null) && isRed(h->right);\r\n Node* x = h->right;\r\n h->right = x->left;\r\n x->left = h;\r\n x->color = x->left->color;\r\n x->left->color = COLOR_RED;\r\n x->size = h->size;\r\n h->size = size(h->left) + size(h->right) + 1;\r\n return x;\r\n }\r\n\r\n // flip the colors of a node and its two children\r\n void flipColors(Node* h) {\r\n // h must have opposite color of its two children\r\n // assert (h != null) && (h->left != null) && (h->right != null);\r\n // assert (!isRed(h) && isRed(h->left) && isRed(h->right))\r\n // || (isRed(h) && !isRed(h->left) && !isRed(h->right));\r\n h->color = !h->color;\r\n h->left->color = !h->left->color;\r\n h->right->color = !h->right->color;\r\n }\r\n\r\n // Assuming that h is red and both h->left and h->left.left\r\n // are black, make h->left or one of its children red.\r\n Node* moveRedLeft(Node* h) {\r\n // assert (h != null);\r\n // assert isRed(h) && !isRed(h->left) && !isRed(h->left.left);\r\n\r\n flipColors(h);\r\n if (isRed(h->right->left)) {\r\n h->right = rotateRight(h->right);\r\n h = rotateLeft(h);\r\n flipColors(h);\r\n }\r\n return h;\r\n }\r\n\r\n // Assuming that h is red and both h->right and h->right.left\r\n // are black, make h->right or one of its children red.\r\n Node* moveRedRight(Node* h) {\r\n // assert (h != null);\r\n // assert isRed(h) && !isRed(h->right) && !isRed(h->right.left);\r\n flipColors(h);\r\n if (isRed(h->left->left)) {\r\n h = rotateRight(h);\r\n flipColors(h);\r\n }\r\n return h;\r\n }\r\n\r\n // restore red-black tree invariant\r\n Node* balance(Node* h) {\r\n // assert (h != null);\r\n\r\n if (isRed(h->right)) h = rotateLeft(h);\r\n if (isRed(h->left) && isRed(h->left->left)) h = rotateRight(h);\r\n if (isRed(h->left) && isRed(h->right)) flipColors(h);\r\n\r\n h->size = size(h->left) + size(h->right) + 1;\r\n return h;\r\n }\r\n\r\n\r\n /***************************************************************************\r\n * Utility functions.\r\n ***************************************************************************/\r\n\r\n /**\r\n * Returns the height of the BST (for debugging).\r\n * @return the height of the BST (a 1-node tree has height 0)\r\n */\r\n int height() {\r\n return height(root);\r\n }\r\n int height(Node* x) {\r\n if (x == nullptr) return -1;\r\n return 1 + std::max(height(x->left), height(x->right));\r\n }\r\n\r\n /***************************************************************************\r\n * Ordered symbol table methods.\r\n ***************************************************************************/\r\n\r\n /**\r\n * Returns the smallest key in the symbol table.\r\n * @return the smallest key in the symbol table\r\n * @throws NoSuchElementException if the symbol table is empty\r\n */\r\n K* min() {\r\n if (isEmpty()) return nullptr;\r\n return min(root).key;\r\n }\r\n\r\n // the smallest key in subtree rooted at x; null if no such key\r\n Node* min(Node* x) {\r\n // assert x != null;\r\n if (x->left == nullptr) return x;\r\n else return min(x->left);\r\n }\r\n\r\n /**\r\n * Returns the largest key in the symbol table.\r\n * @return the largest key in the symbol table\r\n * @throws NoSuchElementException if the symbol table is empty\r\n */\r\n K* max() {\r\n if (isEmpty()) return nullptr;\r\n return max(root).key;\r\n }\r\n\r\n // the largest key in the subtree rooted at x; null if no such key\r\n Node* max(Node* x) {\r\n // assert x != null;\r\n if (x->right == nullptr) return x;\r\n else return max(x->right);\r\n }\r\n\r\n\r\n /**\r\n * Returns the largest key in the symbol table less than or equal to {@code key}.\r\n * @param key the key\r\n * @return the largest key in the symbol table less than or equal to {@code key}\r\n * @throws NoSuchElementException if there is no such key\r\n * @throws IllegalArgumentException if {@code key} is {@code null}\r\n */\r\n K* floor(const K& key) {\r\n if (key == nullptr) return nullptr;\r\n if (isEmpty()) return nullptr;\r\n Node* x = floor(root, key);\r\n if (x == nullptr) return nullptr;\r\n else return x->key;\r\n }\r\n\r\n // the largest key in the subtree rooted at x less than or equal to the given key\r\n Node* floor(Node* x, const K& key) {\r\n if (x == nullptr) return nullptr;\r\n if (key == x->key) return x;\r\n if (key < x->key) return floor(x->left, key);\r\n Node* t = floor(x->right, key);\r\n if (t != nullptr) return t;\r\n else return x;\r\n }\r\n\r\n /**\r\n * Returns the smallest key in the symbol table greater than or equal to {@code key}.\r\n * @param key the key\r\n * @return the smallest key in the symbol table greater than or equal to {@code key}\r\n * @throws NoSuchElementException if there is no such key\r\n * @throws IllegalArgumentException if {@code key} is {@code null}\r\n */\r\n K* ceiling(const K& key) {\r\n if (key == nullptr) return nullptr;\r\n if (isEmpty()) return nullptr;\r\n Node* x = ceiling(root, key);\r\n if (x == nullptr) return nullptr;\r\n else return x->key;\r\n }\r\n\r\n // the smallest key in the subtree rooted at x greater than or equal to the given key\r\n Node* ceiling(Node* x, const K& key) {\r\n if (x == nullptr) return nullptr;\r\n if (key == x->key) return x;\r\n if (x->key < key) return ceiling(x->right, key);\r\n Node* t = ceiling(x->left, key);\r\n if (t != nullptr) return t;\r\n else return x;\r\n }\r\n\r\n /**\r\n * Return the kth smallest key in the symbol table.\r\n * @param k the order statistic\r\n * @return the {@code k}th smallest key in the symbol table\r\n * @throws IllegalArgumentException unless {@code k} is between 0 and\r\n * n1\r\n */\r\n K* select(int k) {\r\n if (k < 0 || k >= size()) {\r\n return nullptr;\r\n }\r\n Node x = select(root, k);\r\n return x->key;\r\n }\r\n\r\n // the key of rank k in the subtree rooted at x\r\n Node* select(Node* x, int k) {\r\n // assert x != null;\r\n // assert k >= 0 && k < size(x);\r\n int t = size(x->left);\r\n if (t > k) return select(x->left, k);\r\n else if (t < k) return select(x->right, k-t-1);\r\n else return x;\r\n }\r\n\r\n /**\r\n * Return the number of keys in the symbol table strictly less than {@code key}.\r\n * @param key the key\r\n * @return the number of keys in the symbol table strictly less than {@code key}\r\n * @throws IllegalArgumentException if {@code key} is {@code null}\r\n */\r\n int rank(const K& key) {\r\n if (key == nullptr) return -1;\r\n return rank(key, root);\r\n }\r\n\r\n // number of keys less than key in the subtree rooted at x\r\n int rank(const K& key, Node* x) {\r\n if (x == nullptr) return 0;\r\n if (key < x->key) return rank(key, x->left);\r\n else if (x->key < key) return 1 + size(x->left) + rank(key, x->right);\r\n else return size(x->left);\r\n }\r\n\r\n /***************************************************************************\r\n * Range count and range search->\r\n ***************************************************************************/\r\n\r\n\r\n /**\r\n * Returns the number of keys in the symbol table in the given range.\r\n *\r\n * @param lo minimum endpoint\r\n * @param hi maximum endpoint\r\n * @return the number of keys in the sybol table between {@code lo}\r\n * (inclusive) and {@code hi} (inclusive)\r\n * @throws IllegalArgumentException if either {@code lo} or {@code hi}\r\n * is {@code null}\r\n */\r\n int size(const K& lo, const K& hi) {\r\n if (lo == nullptr) return 0;\r\n if (hi == nullptr) return 0;\r\n\r\n if (hi < lo) return 0;\r\n if (containsKey(hi)) return rank(hi) - rank(lo) + 1;\r\n else return rank(hi) - rank(lo);\r\n }\r\n\r\n\r\n /***************************************************************************\r\n * Check integrity of red-black tree data structure.\r\n ***************************************************************************/\r\n bool check() {\r\n if (!isBST()) std::cout << \"Not in symmetric order\\n\";\r\n if (!isSizeConsistent()) std::cout << \"Subtree counts not consistent\\n\";\r\n //if (!isRankConsistent()) std::cout << \"Ranks not consistent\\n\";\r\n if (!is23()) std::cout << \"Not a 2-3 tree\\n\";\r\n if (!isBalanced()) std::cout << \"Not balanced\\n\";\r\n return isBST() && isSizeConsistent() && is23() && isBalanced();\r\n }\r\n\r\n // does this binary tree satisfy symmetric order?\r\n // Note: this test also ensures that data structure is a binary tree since order is strict\r\n bool isBST() {\r\n return isBST(root, nullptr, nullptr);\r\n }\r\n\r\n // is the tree rooted at x a BST with all keys strictly between min and max\r\n // (if min or max is null, treat as empty constraint)\r\n // Credit: Bob Dondero's elegant solution\r\n bool isBST(Node* x, K* min, K* max) {\r\n if (x == nullptr) return true;\r\n // TODO: port these two lines\r\n //if (min != nullptr && x->key.compareTo(min) <= 0) return false;\r\n //if (max != nullptr && x->key.compareTo(max) >= 0) return false;\r\n return isBST(x->left, min, x->key) && isBST(x->right, x->key, max);\r\n }\r\n\r\n // are the size fields correct?\r\n bool isSizeConsistent() { return isSizeConsistent(root); }\r\n bool isSizeConsistent(Node* x) {\r\n if (x == nullptr) return true;\r\n if (x->size != size(x->left) + size(x->right) + 1) return false;\r\n return isSizeConsistent(x->left) && isSizeConsistent(x->right);\r\n }\r\n\r\n /*\r\n // check that ranks are consistent\r\n bool isRankConsistent() {\r\n for (int i = 0; i < size(); i++)\r\n if (i != rank(select(i))) return false;\r\n for (K* key : keys())\r\n if (key.compareTo(select(rank(key))) != 0) return false;\r\n return true;\r\n }\r\n */\r\n\r\n // Does the tree have no red right links, and at most one (left)\r\n // red links in a row on any path?\r\n bool is23() { return is23(root); }\r\n bool is23(Node* x) {\r\n if (x == nullptr) return true;\r\n if (isRed(x->right)) return false;\r\n if (x != root && isRed(x) && isRed(x->left))\r\n return false;\r\n return is23(x->left) && is23(x->right);\r\n }\r\n\r\n // do all paths from root to leaf have same number of black edges?\r\n bool isBalanced() {\r\n int black = 0; // number of black links on path from root to min\r\n Node x = root;\r\n while (x != nullptr) {\r\n if (!isRed(x)) black++;\r\n x = x->left;\r\n }\r\n return isBalanced(root, black);\r\n }\r\n\r\n // does every path from the root to a leaf have the given number of black links?\r\n bool isBalanced(Node* x, int black) {\r\n if (x == nullptr) return black == 0;\r\n if (!isRed(x)) black--;\r\n return isBalanced(x->left, black) && isBalanced(x->right, black);\r\n }\r\n\r\n\r\n\r\n // Inserts a key only if it's not already present\r\n bool add(K key, const int tid=0) {\r\n return oflf::updateTx([&] () {\r\n return innerPut(key,key);\r\n });\r\n }\r\n\r\n // Returns true only if the key was present\r\n bool remove(K key, const int tid=0) {\r\n return oflf::updateTx([&] () {\r\n V notused;\r\n bool retval = innerGet(key,notused,false);\r\n if (retval) innerRemove(key);\r\n return retval;\r\n });\r\n }\r\n\r\n bool contains(K key, const int tid=0) {\r\n return oflf::readTx([&] () {\r\n V notused;\r\n return innerGet(key,notused,false);\r\n });\r\n }\r\n\r\n // This is not fully transactionally but it's ok because we use it only on initialization.\r\n // We could make it fully transactionally, but we would have to increase the size of allocation/store logs.\r\n void addAll(K** keys, int size, const int tid=0) {\r\n for (int i = 0; i < size; i++) add(*keys[i], tid);\r\n }\r\n\r\n static std::string className() { return \"OF-LF-RedBlackTree\"; }\r\n\r\n};\r\n\r\n#endif // _TM_RED_BLACK_BST_H_\r\n" }, { "path": "datastructures/treemaps/OFWFRedBlackTree.hpp", "content": "#ifndef _OF_WF_RED_BLACK_BST_H_\r\n#define _OF_WF_RED_BLACK_BST_H_\r\n\r\n#include \r\n#include \r\n#include \r\n\r\n#include \"stms/OneFileWF.hpp\" // This header defines the macros for the STM being compiled\r\n\r\n// Adapted from Java to C++ from the original at http://algs4.cs.princeton.edu/code/edu/princeton/cs/algs4/RedBlackBST.java\r\ntemplate\r\nclass OFWFRedBlackTree {\r\n const int64_t COLOR_RED = 0;\r\n const int64_t COLOR_BLACK = 1;\r\n\r\n struct Node : public ofwf::tmbase {\r\n ofwf::tmtype key;\r\n ofwf::tmtype val;\r\n ofwf::tmtype left {nullptr};\r\n ofwf::tmtype right {nullptr};\r\n ofwf::tmtype color; // color of parent link\r\n ofwf::tmtype size; // subtree count\r\n Node(const K& key, const V& val, int64_t color, int64_t size) : key{key}, val{val}, color{color}, size{size} {}\r\n };\r\n\r\n ofwf::tmtype root {nullptr}; // root of the BST\r\n\r\n inline void assignAndFreeIfNull(ofwf::tmtype& z, Node* w) {\r\n Node* tofree = z;\r\n z = w;\r\n if (w == nullptr) ofwf::tmDelete(tofree);\r\n }\r\n\r\npublic:\r\n /**\r\n * Initializes an empty symbol table.\r\n */\r\n OFWFRedBlackTree(int numThreads=0){ }\r\n\r\n ~OFWFRedBlackTree() {\r\n for (int i = 0; i < 10000; i++) {\r\n ofwf::updateTx([&] () {\r\n if (root == nullptr) return;\r\n deleteMin();\r\n });\r\n }\r\n }\r\n\r\n /***************************************************************************\r\n * Node helper methods.\r\n ***************************************************************************/\r\n // is node x red; false if x is null ?\r\n bool isRed(Node* x) {\r\n if (x == nullptr) return false;\r\n return x->color == COLOR_RED;\r\n }\r\n\r\n // number of node in subtree rooted at x; 0 if x is null\r\n int size(Node* x) {\r\n if (x == nullptr) return 0;\r\n return x->size;\r\n }\r\n\r\n\r\n /**\r\n * Returns the number of key-value pairs in this symbol table.\r\n * @return the number of key-value pairs in this symbol table\r\n */\r\n int size() {\r\n return size(root);\r\n }\r\n\r\n /**\r\n * Is this symbol table empty?\r\n * @return {@code true} if this symbol table is empty and {@code false} otherwise\r\n */\r\n bool isEmpty() {\r\n return root == nullptr;\r\n }\r\n\r\n\r\n /***************************************************************************\r\n * Standard BST search->\r\n ***************************************************************************/\r\n\r\n /**\r\n * Returns the value associated with the given key.\r\n * @param key the key\r\n * @return the value associated with the given key if the key is in the symbol table\r\n * and {@code null} if the key is not in the symbol table\r\n * @throws IllegalArgumentException if {@code key} is {@code null}\r\n */\r\n bool innerGet(K key, V& oldValue, const bool saveOldValue) {\r\n bool found = get(root, key);\r\n if (!found) return false;\r\n //if (saveOldValue) oldValue = *val; // Copy of V\r\n return true;\r\n }\r\n\r\n // value associated with the given key in subtree rooted at x; null if no such key\r\n bool get(Node* x, K& key) {\r\n while (x != nullptr) {\r\n if (key < x->key) x = x->left;\r\n else if (x->key < key) x = x->right;\r\n else return true;\r\n }\r\n return false;\r\n }\r\n\r\n /**\r\n * Does this symbol table contain the given key?\r\n * @param key the key\r\n * @return {@code true} if this symbol table contains {@code key} and\r\n * {@code false} otherwise\r\n * @throws IllegalArgumentException if {@code key} is {@code null}\r\n */\r\n bool containsKey(const K& key) {\r\n return get(key) != nullptr;\r\n }\r\n\r\n /***************************************************************************\r\n * Red-black tree insertion.\r\n ***************************************************************************/\r\n\r\n /**\r\n * Inserts the specified key-value pair into the symbol table, overwriting the old\r\n * value with the new value if the symbol table already contains the specified key.\r\n * Deletes the specified key (and its associated value) from this symbol table\r\n * if the specified value is {@code null}.\r\n *\r\n * @param key the key\r\n * @param val the value\r\n * @throws IllegalArgumentException if {@code key} is {@code null}\r\n */\r\n bool innerPut(const K& key, const V& value) {\r\n \tbool ret = false;\r\n root = put(root, key, value, ret);\r\n root->color = COLOR_BLACK;\r\n return ret;\r\n }\r\n\r\n // insert the key-value pair in the subtree rooted at h\r\n Node* put(Node* h, const K& key, const V& val, bool& ret) {\r\n if (h == nullptr) {\r\n ret = true;\r\n return ofwf::tmNew(key, val, COLOR_RED, 1);\r\n }\r\n if (key < h->key) h->left = put(h->left, key, val, ret);\r\n else if (h->key < key) h->right = put(h->right, key, val, ret);\r\n else h->val = val;\r\n // fix-up any right-leaning links\r\n if (isRed(h->right) && !isRed(h->left)) h = rotateLeft(h);\r\n if (isRed(h->left) && isRed(h->left->left)) h = rotateRight(h);\r\n if (isRed(h->left) && isRed(h->right)) flipColors(h);\r\n h->size = size(h->left) + size(h->right) + 1;\r\n\r\n return h;\r\n }\r\n\r\n /***************************************************************************\r\n * Red-black tree deletion.\r\n ***************************************************************************/\r\n\r\n /**\r\n * Removes the smallest key and associated value from the symbol table.\r\n * @throws NoSuchElementException if the symbol table is empty\r\n */\r\n void deleteMin() {\r\n if (isEmpty()) return;\r\n // if both children of root are black, set root to red\r\n if (!isRed(root->left) && !isRed(root->right))\r\n root->color = COLOR_RED;\r\n assignAndFreeIfNull(root, deleteMin(root));\r\n if (!isEmpty()) root->color = COLOR_BLACK;\r\n // assert check();\r\n }\r\n\r\n // delete the key-value pair with the minimum key rooted at h\r\n Node* deleteMin(Node* h) {\r\n if (h->left == nullptr)\r\n return nullptr;\r\n if (!isRed(h->left) && !isRed(h->left->left))\r\n h = moveRedLeft(h);\r\n assignAndFreeIfNull(h->left, deleteMin(h->left));\r\n return balance(h);\r\n }\r\n\r\n\r\n /**\r\n * Removes the largest key and associated value from the symbol table.\r\n * @throws NoSuchElementException if the symbol table is empty\r\n */\r\n void deleteMax() {\r\n if (isEmpty()) return;\r\n\r\n // if both children of root are black, set root to red\r\n if (!isRed(root->left) && !isRed(root->right))\r\n root->color = COLOR_RED;\r\n\r\n root = deleteMax(root);\r\n if (!isEmpty()) root->color = COLOR_BLACK;\r\n // assert check();\r\n }\r\n\r\n // delete the key-value pair with the maximum key rooted at h\r\n Node* deleteMax(Node* h) {\r\n if (isRed(h->left))\r\n h = rotateRight(h);\r\n\r\n if (h->right == nullptr)\r\n return nullptr;\r\n\r\n if (!isRed(h->right) && !isRed(h->right->left))\r\n h = moveRedRight(h);\r\n\r\n h->right = deleteMax(h->right);\r\n\r\n return balance(h);\r\n }\r\n\r\n /**\r\n * Removes the specified key and its associated value from this symbol table\r\n * (if the key is in this symbol table).\r\n *\r\n * @param key the key\r\n */\r\n void innerRemove(K key) {\r\n // if both children of root are black, set root to red\r\n if (!isRed(root->left) && !isRed(root->right)) root->color = COLOR_RED;\r\n assignAndFreeIfNull(root, deleteKey(root, key));\r\n if (!isEmpty()) root->color = COLOR_BLACK;\r\n // assert check();\r\n }\r\n\r\n // delete the key-value pair with the given key rooted at h\r\n Node* deleteKey(Node* h, const K& key) {\r\n // assert get(h, key) != null;\r\n if (key < h->key) {\r\n if (!isRed(h->left) && !isRed(h->left->left)) {\r\n h = moveRedLeft(h);\r\n }\r\n assignAndFreeIfNull(h->left, deleteKey(h->left, key));\r\n } else {\r\n if (isRed(h->left)) {\r\n h = rotateRight(h);\r\n }\r\n if (key == h->key && (h->right == nullptr)) {\r\n return nullptr;\r\n }\r\n if (!isRed(h->right) && !isRed(h->right->left)) {\r\n h = moveRedRight(h);\r\n }\r\n if (key == h->key) {\r\n Node* x = min(h->right);\r\n h->key = x->key;\r\n h->val = x->val;\r\n // h->val = get(h->right, min(h->right).key);\r\n // h->key = min(h->right).key;\r\n assignAndFreeIfNull(h->right, deleteMin(h->right));\r\n } else {\r\n assignAndFreeIfNull(h->right, deleteKey(h->right, key));\r\n }\r\n }\r\n return balance(h);\r\n }\r\n\r\n /***************************************************************************\r\n * Red-black tree helper functions.\r\n ***************************************************************************/\r\n\r\n // make a left-leaning link lean to the right\r\n Node* rotateRight(Node* h) {\r\n // assert (h != null) && isRed(h->left);\r\n Node* x = h->left;\r\n h->left = x->right;\r\n x->right = h;\r\n x->color = x->right->color;\r\n x->right->color = COLOR_RED;\r\n x->size = h->size;\r\n h->size = size(h->left) + size(h->right) + 1;\r\n return x;\r\n }\r\n\r\n // make a right-leaning link lean to the left\r\n Node* rotateLeft(Node* h) {\r\n // assert (h != null) && isRed(h->right);\r\n Node* x = h->right;\r\n h->right = x->left;\r\n x->left = h;\r\n x->color = x->left->color;\r\n x->left->color = COLOR_RED;\r\n x->size = h->size;\r\n h->size = size(h->left) + size(h->right) + 1;\r\n return x;\r\n }\r\n\r\n // flip the colors of a node and its two children\r\n void flipColors(Node* h) {\r\n // h must have opposite color of its two children\r\n // assert (h != null) && (h->left != null) && (h->right != null);\r\n // assert (!isRed(h) && isRed(h->left) && isRed(h->right))\r\n // || (isRed(h) && !isRed(h->left) && !isRed(h->right));\r\n h->color = !h->color;\r\n h->left->color = !h->left->color;\r\n h->right->color = !h->right->color;\r\n }\r\n\r\n // Assuming that h is red and both h->left and h->left.left\r\n // are black, make h->left or one of its children red.\r\n Node* moveRedLeft(Node* h) {\r\n // assert (h != null);\r\n // assert isRed(h) && !isRed(h->left) && !isRed(h->left.left);\r\n\r\n flipColors(h);\r\n if (isRed(h->right->left)) {\r\n h->right = rotateRight(h->right);\r\n h = rotateLeft(h);\r\n flipColors(h);\r\n }\r\n return h;\r\n }\r\n\r\n // Assuming that h is red and both h->right and h->right.left\r\n // are black, make h->right or one of its children red.\r\n Node* moveRedRight(Node* h) {\r\n // assert (h != null);\r\n // assert isRed(h) && !isRed(h->right) && !isRed(h->right.left);\r\n flipColors(h);\r\n if (isRed(h->left->left)) {\r\n h = rotateRight(h);\r\n flipColors(h);\r\n }\r\n return h;\r\n }\r\n\r\n // restore red-black tree invariant\r\n Node* balance(Node* h) {\r\n // assert (h != null);\r\n\r\n if (isRed(h->right)) h = rotateLeft(h);\r\n if (isRed(h->left) && isRed(h->left->left)) h = rotateRight(h);\r\n if (isRed(h->left) && isRed(h->right)) flipColors(h);\r\n\r\n h->size = size(h->left) + size(h->right) + 1;\r\n return h;\r\n }\r\n\r\n\r\n /***************************************************************************\r\n * Utility functions.\r\n ***************************************************************************/\r\n\r\n /**\r\n * Returns the height of the BST (for debugging).\r\n * @return the height of the BST (a 1-node tree has height 0)\r\n */\r\n int height() {\r\n return height(root);\r\n }\r\n int height(Node* x) {\r\n if (x == nullptr) return -1;\r\n return 1 + std::max(height(x->left), height(x->right));\r\n }\r\n\r\n /***************************************************************************\r\n * Ordered symbol table methods.\r\n ***************************************************************************/\r\n\r\n /**\r\n * Returns the smallest key in the symbol table.\r\n * @return the smallest key in the symbol table\r\n * @throws NoSuchElementException if the symbol table is empty\r\n */\r\n K* min() {\r\n if (isEmpty()) return nullptr;\r\n return min(root).key;\r\n }\r\n\r\n // the smallest key in subtree rooted at x; null if no such key\r\n Node* min(Node* x) {\r\n // assert x != null;\r\n if (x->left == nullptr) return x;\r\n else return min(x->left);\r\n }\r\n\r\n /**\r\n * Returns the largest key in the symbol table.\r\n * @return the largest key in the symbol table\r\n * @throws NoSuchElementException if the symbol table is empty\r\n */\r\n K* max() {\r\n if (isEmpty()) return nullptr;\r\n return max(root).key;\r\n }\r\n\r\n // the largest key in the subtree rooted at x; null if no such key\r\n Node* max(Node* x) {\r\n // assert x != null;\r\n if (x->right == nullptr) return x;\r\n else return max(x->right);\r\n }\r\n\r\n\r\n /**\r\n * Returns the largest key in the symbol table less than or equal to {@code key}.\r\n * @param key the key\r\n * @return the largest key in the symbol table less than or equal to {@code key}\r\n * @throws NoSuchElementException if there is no such key\r\n * @throws IllegalArgumentException if {@code key} is {@code null}\r\n */\r\n K* floor(const K& key) {\r\n if (key == nullptr) return nullptr;\r\n if (isEmpty()) return nullptr;\r\n Node* x = floor(root, key);\r\n if (x == nullptr) return nullptr;\r\n else return x->key;\r\n }\r\n\r\n // the largest key in the subtree rooted at x less than or equal to the given key\r\n Node* floor(Node* x, const K& key) {\r\n if (x == nullptr) return nullptr;\r\n if (key == x->key) return x;\r\n if (key < x->key) return floor(x->left, key);\r\n Node* t = floor(x->right, key);\r\n if (t != nullptr) return t;\r\n else return x;\r\n }\r\n\r\n /**\r\n * Returns the smallest key in the symbol table greater than or equal to {@code key}.\r\n * @param key the key\r\n * @return the smallest key in the symbol table greater than or equal to {@code key}\r\n * @throws NoSuchElementException if there is no such key\r\n * @throws IllegalArgumentException if {@code key} is {@code null}\r\n */\r\n K* ceiling(const K& key) {\r\n if (key == nullptr) return nullptr;\r\n if (isEmpty()) return nullptr;\r\n Node* x = ceiling(root, key);\r\n if (x == nullptr) return nullptr;\r\n else return x->key;\r\n }\r\n\r\n // the smallest key in the subtree rooted at x greater than or equal to the given key\r\n Node* ceiling(Node* x, const K& key) {\r\n if (x == nullptr) return nullptr;\r\n if (key == x->key) return x;\r\n if (x->key < key) return ceiling(x->right, key);\r\n Node* t = ceiling(x->left, key);\r\n if (t != nullptr) return t;\r\n else return x;\r\n }\r\n\r\n /**\r\n * Return the kth smallest key in the symbol table.\r\n * @param k the order statistic\r\n * @return the {@code k}th smallest key in the symbol table\r\n * @throws IllegalArgumentException unless {@code k} is between 0 and\r\n * n1\r\n */\r\n K* select(int k) {\r\n if (k < 0 || k >= size()) {\r\n return nullptr;\r\n }\r\n Node x = select(root, k);\r\n return x->key;\r\n }\r\n\r\n // the key of rank k in the subtree rooted at x\r\n Node* select(Node* x, int k) {\r\n // assert x != null;\r\n // assert k >= 0 && k < size(x);\r\n int t = size(x->left);\r\n if (t > k) return select(x->left, k);\r\n else if (t < k) return select(x->right, k-t-1);\r\n else return x;\r\n }\r\n\r\n /**\r\n * Return the number of keys in the symbol table strictly less than {@code key}.\r\n * @param key the key\r\n * @return the number of keys in the symbol table strictly less than {@code key}\r\n * @throws IllegalArgumentException if {@code key} is {@code null}\r\n */\r\n int rank(const K& key) {\r\n if (key == nullptr) return -1;\r\n return rank(key, root);\r\n }\r\n\r\n // number of keys less than key in the subtree rooted at x\r\n int rank(const K& key, Node* x) {\r\n if (x == nullptr) return 0;\r\n if (key < x->key) return rank(key, x->left);\r\n else if (x->key < key) return 1 + size(x->left) + rank(key, x->right);\r\n else return size(x->left);\r\n }\r\n\r\n /***************************************************************************\r\n * Range count and range search->\r\n ***************************************************************************/\r\n\r\n\r\n /**\r\n * Returns the number of keys in the symbol table in the given range.\r\n *\r\n * @param lo minimum endpoint\r\n * @param hi maximum endpoint\r\n * @return the number of keys in the sybol table between {@code lo}\r\n * (inclusive) and {@code hi} (inclusive)\r\n * @throws IllegalArgumentException if either {@code lo} or {@code hi}\r\n * is {@code null}\r\n */\r\n int size(const K& lo, const K& hi) {\r\n if (lo == nullptr) return 0;\r\n if (hi == nullptr) return 0;\r\n\r\n if (hi < lo) return 0;\r\n if (containsKey(hi)) return rank(hi) - rank(lo) + 1;\r\n else return rank(hi) - rank(lo);\r\n }\r\n\r\n\r\n /***************************************************************************\r\n * Check integrity of red-black tree data structure.\r\n ***************************************************************************/\r\n bool check() {\r\n if (!isBST()) std::cout << \"Not in symmetric order\\n\";\r\n if (!isSizeConsistent()) std::cout << \"Subtree counts not consistent\\n\";\r\n //if (!isRankConsistent()) std::cout << \"Ranks not consistent\\n\";\r\n if (!is23()) std::cout << \"Not a 2-3 tree\\n\";\r\n if (!isBalanced()) std::cout << \"Not balanced\\n\";\r\n return isBST() && isSizeConsistent() && is23() && isBalanced();\r\n }\r\n\r\n // does this binary tree satisfy symmetric order?\r\n // Note: this test also ensures that data structure is a binary tree since order is strict\r\n bool isBST() {\r\n return isBST(root, nullptr, nullptr);\r\n }\r\n\r\n // is the tree rooted at x a BST with all keys strictly between min and max\r\n // (if min or max is null, treat as empty constraint)\r\n // Credit: Bob Dondero's elegant solution\r\n bool isBST(Node* x, K* min, K* max) {\r\n if (x == nullptr) return true;\r\n // TODO: port these two lines\r\n //if (min != nullptr && x->key.compareTo(min) <= 0) return false;\r\n //if (max != nullptr && x->key.compareTo(max) >= 0) return false;\r\n return isBST(x->left, min, x->key) && isBST(x->right, x->key, max);\r\n }\r\n\r\n // are the size fields correct?\r\n bool isSizeConsistent() { return isSizeConsistent(root); }\r\n bool isSizeConsistent(Node* x) {\r\n if (x == nullptr) return true;\r\n if (x->size != size(x->left) + size(x->right) + 1) return false;\r\n return isSizeConsistent(x->left) && isSizeConsistent(x->right);\r\n }\r\n\r\n /*\r\n // check that ranks are consistent\r\n bool isRankConsistent() {\r\n for (int i = 0; i < size(); i++)\r\n if (i != rank(select(i))) return false;\r\n for (K* key : keys())\r\n if (key.compareTo(select(rank(key))) != 0) return false;\r\n return true;\r\n }\r\n */\r\n\r\n // Does the tree have no red right links, and at most one (left)\r\n // red links in a row on any path?\r\n bool is23() { return is23(root); }\r\n bool is23(Node* x) {\r\n if (x == nullptr) return true;\r\n if (isRed(x->right)) return false;\r\n if (x != root && isRed(x) && isRed(x->left))\r\n return false;\r\n return is23(x->left) && is23(x->right);\r\n }\r\n\r\n // do all paths from root to leaf have same number of black edges?\r\n bool isBalanced() {\r\n int black = 0; // number of black links on path from root to min\r\n Node x = root;\r\n while (x != nullptr) {\r\n if (!isRed(x)) black++;\r\n x = x->left;\r\n }\r\n return isBalanced(root, black);\r\n }\r\n\r\n // does every path from the root to a leaf have the given number of black links?\r\n bool isBalanced(Node* x, int black) {\r\n if (x == nullptr) return black == 0;\r\n if (!isRed(x)) black--;\r\n return isBalanced(x->left, black) && isBalanced(x->right, black);\r\n }\r\n\r\n\r\n\r\n // Inserts a key only if it's not already present\r\n bool add(K key, const int tid=0) {\r\n return ofwf::updateTx([=] () {\r\n return innerPut(key,key);\r\n });\r\n }\r\n\r\n // Returns true only if the key was present\r\n bool remove(K key, const int tid=0) {\r\n return ofwf::updateTx([=] () {\r\n V notused;\r\n bool retval = innerGet(key,notused,false);\r\n if (retval) innerRemove(key);\r\n return retval;\r\n });\r\n }\r\n\r\n bool contains(K key, const int tid=0) {\r\n return ofwf::readTx([=] () {\r\n V notused;\r\n return innerGet(key,notused,false);\r\n });\r\n }\r\n\r\n void addAll(K** keys, int size, const int tid=0) {\r\n for (int i = 0; i < size; i++) add(*keys[i], tid);\r\n }\r\n\r\n static std::string className() { return \"OF-WF-RedBlackTree\"; }\r\n\r\n};\r\n\r\n#endif // _OF_WF_RED_BLACK_BST_H_\r\n" }, { "path": "datastructures/treemaps/TinySTMRedBlackTree.hpp", "content": "#ifndef _TINY_STM_RED_BLACK_BST_H_\r\n#define _TINY_STM_RED_BLACK_BST_H_\r\n\r\n#include \r\n#include \r\n#include \r\n\r\n#include \"stms/TinySTM.hpp\"\r\n\r\n// Adapted from Java to C++ from the original at http://algs4.cs.princeton.edu/code/edu/princeton/cs/algs4/RedBlackBST.java\r\ntemplate\r\nclass TinySTMRedBlackTree {\r\n const int64_t COLOR_RED = 0;\r\n const int64_t COLOR_BLACK = 1;\r\n\r\n struct Node {\r\n tinystm::tmtype key;\r\n tinystm::tmtype val;\r\n tinystm::tmtype left {nullptr};\r\n tinystm::tmtype right {nullptr};\r\n tinystm::tmtype color; // color of parent link\r\n tinystm::tmtype size; // subtree count\r\n Node(const K& key, const V& val, int64_t color, int64_t size) : key{key}, val{val}, color{color}, size{size} {}\r\n };\r\n\r\n tinystm::tmtype root {nullptr}; // root of the BST\r\n\r\n inline void assignAndFreeIfNull(tinystm::tmtype& z, Node* w) {\r\n Node* tofree = z;\r\n z = w;\r\n if (w == nullptr) tinystm::tmDelete(tofree);\r\n }\r\n\r\npublic:\r\n /**\r\n * Initializes an empty symbol table.\r\n */\r\n TinySTMRedBlackTree(int maxThreads=0){ }\r\n\r\n ~TinySTMRedBlackTree() {\r\n // The transaction log is not enough to delete everything if there are too many, so we delete 1000 per transaction\r\n for (int i = 0; i < 1000; i++) {\r\n tinystm::updateTx([&] () {\r\n if (root == nullptr) return true;\r\n deleteMin();\r\n return true;\r\n });\r\n }\r\n }\r\n\r\n /***************************************************************************\r\n * Node helper methods.\r\n ***************************************************************************/\r\n // is node x red; false if x is null ?\r\n bool isRed(Node* x) {\r\n if (x == nullptr) return false;\r\n return x->color == COLOR_RED;\r\n }\r\n\r\n // number of node in subtree rooted at x; 0 if x is null\r\n int size(Node* x) {\r\n if (x == nullptr) return 0;\r\n return x->size;\r\n }\r\n\r\n\r\n /**\r\n * Returns the number of key-value pairs in this symbol table.\r\n * @return the number of key-value pairs in this symbol table\r\n */\r\n int size() {\r\n return size(root);\r\n }\r\n\r\n /**\r\n * Is this symbol table empty?\r\n * @return {@code true} if this symbol table is empty and {@code false} otherwise\r\n */\r\n bool isEmpty() {\r\n return root == nullptr;\r\n }\r\n\r\n\r\n /***************************************************************************\r\n * Standard BST search->\r\n ***************************************************************************/\r\n\r\n /**\r\n * Returns the value associated with the given key.\r\n * @param key the key\r\n * @return the value associated with the given key if the key is in the symbol table\r\n * and {@code null} if the key is not in the symbol table\r\n * @throws IllegalArgumentException if {@code key} is {@code null}\r\n */\r\n bool innerGet(K& key, V& oldValue, const bool saveOldValue) {\r\n bool found = get(root, key);\r\n if (!found) return false;\r\n //if (saveOldValue) oldValue = *val; // Copy of V\r\n return true;\r\n }\r\n\r\n // value associated with the given key in subtree rooted at x; null if no such key\r\n bool get(Node* x, K& key) {\r\n while (x != nullptr) {\r\n if (key < x->key) x = x->left;\r\n else if (x->key < key) x = x->right;\r\n else return true;\r\n }\r\n return false;\r\n }\r\n\r\n /**\r\n * Does this symbol table contain the given key?\r\n * @param key the key\r\n * @return {@code true} if this symbol table contains {@code key} and\r\n * {@code false} otherwise\r\n * @throws IllegalArgumentException if {@code key} is {@code null}\r\n */\r\n bool containsKey(const K& key) {\r\n return get(key) != nullptr;\r\n }\r\n\r\n /***************************************************************************\r\n * Red-black tree insertion.\r\n ***************************************************************************/\r\n\r\n /**\r\n * Inserts the specified key-value pair into the symbol table, overwriting the old\r\n * value with the new value if the symbol table already contains the specified key.\r\n * Deletes the specified key (and its associated value) from this symbol table\r\n * if the specified value is {@code null}.\r\n *\r\n * @param key the key\r\n * @param val the value\r\n * @throws IllegalArgumentException if {@code key} is {@code null}\r\n */\r\n bool innerPut(const K& key, const V& value) {\r\n \tbool ret = false;\r\n root = put(root, key, value, ret);\r\n root->color = COLOR_BLACK;\r\n return ret;\r\n }\r\n\r\n // insert the key-value pair in the subtree rooted at h\r\n Node* put(Node* h, const K& key, const V& val, bool& ret) {\r\n if (h == nullptr) {\r\n \tret = true;\r\n \treturn tinystm::tmNew(key, val, COLOR_RED, 1);\r\n }\r\n if (key < h->key) h->left = put(h->left, key, val, ret);\r\n else if (h->key < key) h->right = put(h->right, key, val, ret);\r\n else h->val = val;\r\n // fix-up any right-leaning links\r\n if (isRed(h->right) && !isRed(h->left)) h = rotateLeft(h);\r\n if (isRed(h->left) && isRed(h->left->left)) h = rotateRight(h);\r\n if (isRed(h->left) && isRed(h->right)) flipColors(h);\r\n h->size = size(h->left) + size(h->right) + 1;\r\n\r\n return h;\r\n }\r\n\r\n /***************************************************************************\r\n * Red-black tree deletion.\r\n ***************************************************************************/\r\n\r\n /**\r\n * Removes the smallest key and associated value from the symbol table.\r\n * @throws NoSuchElementException if the symbol table is empty\r\n */\r\n void deleteMin() {\r\n if (isEmpty()) return;\r\n // if both children of root are black, set root to red\r\n if (!isRed(root->left) && !isRed(root->right))\r\n root->color = COLOR_RED;\r\n assignAndFreeIfNull(root, deleteMin(root));\r\n if (!isEmpty()) root->color = COLOR_BLACK;\r\n // assert check();\r\n }\r\n\r\n // delete the key-value pair with the minimum key rooted at h\r\n Node* deleteMin(Node* h) {\r\n if (h->left == nullptr)\r\n return nullptr;\r\n if (!isRed(h->left) && !isRed(h->left->left))\r\n h = moveRedLeft(h);\r\n assignAndFreeIfNull(h->left, deleteMin(h->left));\r\n return balance(h);\r\n }\r\n\r\n\r\n /**\r\n * Removes the largest key and associated value from the symbol table.\r\n * @throws NoSuchElementException if the symbol table is empty\r\n */\r\n void deleteMax() {\r\n if (isEmpty()) return;\r\n\r\n // if both children of root are black, set root to red\r\n if (!isRed(root->left) && !isRed(root->right))\r\n root->color = COLOR_RED;\r\n\r\n root = deleteMax(root);\r\n if (!isEmpty()) root->color = COLOR_BLACK;\r\n // assert check();\r\n }\r\n\r\n // delete the key-value pair with the maximum key rooted at h\r\n Node* deleteMax(Node* h) {\r\n if (isRed(h->left))\r\n h = rotateRight(h);\r\n\r\n if (h->right == nullptr)\r\n return nullptr;\r\n\r\n if (!isRed(h->right) && !isRed(h->right->left))\r\n h = moveRedRight(h);\r\n\r\n h->right = deleteMax(h->right);\r\n\r\n return balance(h);\r\n }\r\n\r\n /**\r\n * Removes the specified key and its associated value from this symbol table\r\n * (if the key is in this symbol table).\r\n *\r\n * @param key the key\r\n */\r\n void innerRemove(const K& key) {\r\n // if both children of root are black, set root to red\r\n if (!isRed(root->left) && !isRed(root->right)) root->color = COLOR_RED;\r\n assignAndFreeIfNull(root, deleteKey(root, key));\r\n if (!isEmpty()) root->color = COLOR_BLACK;\r\n // assert check();\r\n }\r\n\r\n // delete the key-value pair with the given key rooted at h\r\n Node* deleteKey(Node* h, const K& key) {\r\n // assert get(h, key) != null;\r\n if (key < h->key) {\r\n if (!isRed(h->left) && !isRed(h->left->left)) {\r\n h = moveRedLeft(h);\r\n }\r\n assignAndFreeIfNull(h->left, deleteKey(h->left, key));\r\n } else {\r\n if (isRed(h->left)) {\r\n h = rotateRight(h);\r\n }\r\n if (key == h->key && (h->right == nullptr)) {\r\n return nullptr;\r\n }\r\n if (!isRed(h->right) && !isRed(h->right->left)) {\r\n h = moveRedRight(h);\r\n }\r\n if (key == h->key) {\r\n Node* x = min(h->right);\r\n h->key = x->key;\r\n h->val = x->val;\r\n // h->val = get(h->right, min(h->right).key);\r\n // h->key = min(h->right).key;\r\n assignAndFreeIfNull(h->right, deleteMin(h->right));\r\n } else {\r\n assignAndFreeIfNull(h->right, deleteKey(h->right, key));\r\n }\r\n }\r\n return balance(h);\r\n }\r\n\r\n /***************************************************************************\r\n * Red-black tree helper functions.\r\n ***************************************************************************/\r\n\r\n // make a left-leaning link lean to the right\r\n Node* rotateRight(Node* h) {\r\n // assert (h != null) && isRed(h->left);\r\n Node* x = h->left;\r\n h->left = x->right;\r\n x->right = h;\r\n x->color = x->right->color;\r\n x->right->color = COLOR_RED;\r\n x->size = h->size;\r\n h->size = size(h->left) + size(h->right) + 1;\r\n return x;\r\n }\r\n\r\n // make a right-leaning link lean to the left\r\n Node* rotateLeft(Node* h) {\r\n // assert (h != null) && isRed(h->right);\r\n Node* x = h->right;\r\n h->right = x->left;\r\n x->left = h;\r\n x->color = x->left->color;\r\n x->left->color = COLOR_RED;\r\n x->size = h->size;\r\n h->size = size(h->left) + size(h->right) + 1;\r\n return x;\r\n }\r\n\r\n // flip the colors of a node and its two children\r\n void flipColors(Node* h) {\r\n // h must have opposite color of its two children\r\n // assert (h != null) && (h->left != null) && (h->right != null);\r\n // assert (!isRed(h) && isRed(h->left) && isRed(h->right))\r\n // || (isRed(h) && !isRed(h->left) && !isRed(h->right));\r\n h->color = !h->color;\r\n h->left->color = !h->left->color;\r\n h->right->color = !h->right->color;\r\n }\r\n\r\n // Assuming that h is red and both h->left and h->left.left\r\n // are black, make h->left or one of its children red.\r\n Node* moveRedLeft(Node* h) {\r\n // assert (h != null);\r\n // assert isRed(h) && !isRed(h->left) && !isRed(h->left.left);\r\n\r\n flipColors(h);\r\n if (isRed(h->right->left)) {\r\n h->right = rotateRight(h->right);\r\n h = rotateLeft(h);\r\n flipColors(h);\r\n }\r\n return h;\r\n }\r\n\r\n // Assuming that h is red and both h->right and h->right.left\r\n // are black, make h->right or one of its children red.\r\n Node* moveRedRight(Node* h) {\r\n // assert (h != null);\r\n // assert isRed(h) && !isRed(h->right) && !isRed(h->right.left);\r\n flipColors(h);\r\n if (isRed(h->left->left)) {\r\n h = rotateRight(h);\r\n flipColors(h);\r\n }\r\n return h;\r\n }\r\n\r\n // restore red-black tree invariant\r\n Node* balance(Node* h) {\r\n // assert (h != null);\r\n\r\n if (isRed(h->right)) h = rotateLeft(h);\r\n if (isRed(h->left) && isRed(h->left->left)) h = rotateRight(h);\r\n if (isRed(h->left) && isRed(h->right)) flipColors(h);\r\n\r\n h->size = size(h->left) + size(h->right) + 1;\r\n return h;\r\n }\r\n\r\n\r\n /***************************************************************************\r\n * Utility functions.\r\n ***************************************************************************/\r\n\r\n /**\r\n * Returns the height of the BST (for debugging).\r\n * @return the height of the BST (a 1-node tree has height 0)\r\n */\r\n int height() {\r\n return height(root);\r\n }\r\n int height(Node* x) {\r\n if (x == nullptr) return -1;\r\n return 1 + std::max(height(x->left), height(x->right));\r\n }\r\n\r\n /***************************************************************************\r\n * Ordered symbol table methods.\r\n ***************************************************************************/\r\n\r\n /**\r\n * Returns the smallest key in the symbol table.\r\n * @return the smallest key in the symbol table\r\n * @throws NoSuchElementException if the symbol table is empty\r\n */\r\n K* min() {\r\n if (isEmpty()) return nullptr;\r\n return min(root).key;\r\n }\r\n\r\n // the smallest key in subtree rooted at x; null if no such key\r\n Node* min(Node* x) {\r\n // assert x != null;\r\n if (x->left == nullptr) return x;\r\n else return min(x->left);\r\n }\r\n\r\n /**\r\n * Returns the largest key in the symbol table.\r\n * @return the largest key in the symbol table\r\n * @throws NoSuchElementException if the symbol table is empty\r\n */\r\n K* max() {\r\n if (isEmpty()) return nullptr;\r\n return max(root).key;\r\n }\r\n\r\n // the largest key in the subtree rooted at x; null if no such key\r\n Node* max(Node* x) {\r\n // assert x != null;\r\n if (x->right == nullptr) return x;\r\n else return max(x->right);\r\n }\r\n\r\n\r\n /**\r\n * Returns the largest key in the symbol table less than or equal to {@code key}.\r\n * @param key the key\r\n * @return the largest key in the symbol table less than or equal to {@code key}\r\n * @throws NoSuchElementException if there is no such key\r\n * @throws IllegalArgumentException if {@code key} is {@code null}\r\n */\r\n K* floor(const K& key) {\r\n if (key == nullptr) return nullptr;\r\n if (isEmpty()) return nullptr;\r\n Node* x = floor(root, key);\r\n if (x == nullptr) return nullptr;\r\n else return x->key;\r\n }\r\n\r\n // the largest key in the subtree rooted at x less than or equal to the given key\r\n Node* floor(Node* x, const K& key) {\r\n if (x == nullptr) return nullptr;\r\n if (key == x->key) return x;\r\n if (key < x->key) return floor(x->left, key);\r\n Node* t = floor(x->right, key);\r\n if (t != nullptr) return t;\r\n else return x;\r\n }\r\n\r\n /**\r\n * Returns the smallest key in the symbol table greater than or equal to {@code key}.\r\n * @param key the key\r\n * @return the smallest key in the symbol table greater than or equal to {@code key}\r\n * @throws NoSuchElementException if there is no such key\r\n * @throws IllegalArgumentException if {@code key} is {@code null}\r\n */\r\n K* ceiling(const K& key) {\r\n if (key == nullptr) return nullptr;\r\n if (isEmpty()) return nullptr;\r\n Node* x = ceiling(root, key);\r\n if (x == nullptr) return nullptr;\r\n else return x->key;\r\n }\r\n\r\n // the smallest key in the subtree rooted at x greater than or equal to the given key\r\n Node* ceiling(Node* x, const K& key) {\r\n if (x == nullptr) return nullptr;\r\n if (key == x->key) return x;\r\n if (x->key < key) return ceiling(x->right, key);\r\n Node* t = ceiling(x->left, key);\r\n if (t != nullptr) return t;\r\n else return x;\r\n }\r\n\r\n /**\r\n * Return the kth smallest key in the symbol table.\r\n * @param k the order statistic\r\n * @return the {@code k}th smallest key in the symbol table\r\n * @throws IllegalArgumentException unless {@code k} is between 0 and\r\n * n1\r\n */\r\n K* select(int k) {\r\n if (k < 0 || k >= size()) {\r\n return nullptr;\r\n }\r\n Node x = select(root, k);\r\n return x->key;\r\n }\r\n\r\n // the key of rank k in the subtree rooted at x\r\n Node* select(Node* x, int k) {\r\n // assert x != null;\r\n // assert k >= 0 && k < size(x);\r\n int t = size(x->left);\r\n if (t > k) return select(x->left, k);\r\n else if (t < k) return select(x->right, k-t-1);\r\n else return x;\r\n }\r\n\r\n /**\r\n * Return the number of keys in the symbol table strictly less than {@code key}.\r\n * @param key the key\r\n * @return the number of keys in the symbol table strictly less than {@code key}\r\n * @throws IllegalArgumentException if {@code key} is {@code null}\r\n */\r\n int rank(const K& key) {\r\n if (key == nullptr) return -1;\r\n return rank(key, root);\r\n }\r\n\r\n // number of keys less than key in the subtree rooted at x\r\n int rank(const K& key, Node* x) {\r\n if (x == nullptr) return 0;\r\n if (key < x->key) return rank(key, x->left);\r\n else if (x->key < key) return 1 + size(x->left) + rank(key, x->right);\r\n else return size(x->left);\r\n }\r\n\r\n /***************************************************************************\r\n * Range count and range search->\r\n ***************************************************************************/\r\n\r\n\r\n /**\r\n * Returns the number of keys in the symbol table in the given range.\r\n *\r\n * @param lo minimum endpoint\r\n * @param hi maximum endpoint\r\n * @return the number of keys in the sybol table between {@code lo}\r\n * (inclusive) and {@code hi} (inclusive)\r\n * @throws IllegalArgumentException if either {@code lo} or {@code hi}\r\n * is {@code null}\r\n */\r\n int size(const K& lo, const K& hi) {\r\n if (lo == nullptr) return 0;\r\n if (hi == nullptr) return 0;\r\n\r\n if (hi < lo) return 0;\r\n if (containsKey(hi)) return rank(hi) - rank(lo) + 1;\r\n else return rank(hi) - rank(lo);\r\n }\r\n\r\n\r\n /***************************************************************************\r\n * Check integrity of red-black tree data structure.\r\n ***************************************************************************/\r\n bool check() {\r\n if (!isBST()) std::cout << \"Not in symmetric order\\n\";\r\n if (!isSizeConsistent()) std::cout << \"Subtree counts not consistent\\n\";\r\n //if (!isRankConsistent()) std::cout << \"Ranks not consistent\\n\";\r\n if (!is23()) std::cout << \"Not a 2-3 tree\\n\";\r\n if (!isBalanced()) std::cout << \"Not balanced\\n\";\r\n return isBST() && isSizeConsistent() && is23() && isBalanced();\r\n }\r\n\r\n // does this binary tree satisfy symmetric order?\r\n // Note: this test also ensures that data structure is a binary tree since order is strict\r\n bool isBST() {\r\n return isBST(root, nullptr, nullptr);\r\n }\r\n\r\n // is the tree rooted at x a BST with all keys strictly between min and max\r\n // (if min or max is null, treat as empty constraint)\r\n // Credit: Bob Dondero's elegant solution\r\n bool isBST(Node* x, K* min, K* max) {\r\n if (x == nullptr) return true;\r\n // TODO: port these two lines\r\n //if (min != nullptr && x->key.compareTo(min) <= 0) return false;\r\n //if (max != nullptr && x->key.compareTo(max) >= 0) return false;\r\n return isBST(x->left, min, x->key) && isBST(x->right, x->key, max);\r\n }\r\n\r\n // are the size fields correct?\r\n bool isSizeConsistent() { return isSizeConsistent(root); }\r\n bool isSizeConsistent(Node* x) {\r\n if (x == nullptr) return true;\r\n if (x->size != size(x->left) + size(x->right) + 1) return false;\r\n return isSizeConsistent(x->left) && isSizeConsistent(x->right);\r\n }\r\n\r\n /*\r\n // check that ranks are consistent\r\n bool isRankConsistent() {\r\n for (int i = 0; i < size(); i++)\r\n if (i != rank(select(i))) return false;\r\n for (K* key : keys())\r\n if (key.compareTo(select(rank(key))) != 0) return false;\r\n return true;\r\n }\r\n */\r\n\r\n // Does the tree have no red right links, and at most one (left)\r\n // red links in a row on any path?\r\n bool is23() { return is23(root); }\r\n bool is23(Node* x) {\r\n if (x == nullptr) return true;\r\n if (isRed(x->right)) return false;\r\n if (x != root && isRed(x) && isRed(x->left))\r\n return false;\r\n return is23(x->left) && is23(x->right);\r\n }\r\n\r\n // do all paths from root to leaf have same number of black edges?\r\n bool isBalanced() {\r\n int black = 0; // number of black links on path from root to min\r\n Node x = root;\r\n while (x != nullptr) {\r\n if (!isRed(x)) black++;\r\n x = x->left;\r\n }\r\n return isBalanced(root, black);\r\n }\r\n\r\n // does every path from the root to a leaf have the given number of black links?\r\n bool isBalanced(Node* x, int black) {\r\n if (x == nullptr) return black == 0;\r\n if (!isRed(x)) black--;\r\n return isBalanced(x->left, black) && isBalanced(x->right, black);\r\n }\r\n\r\n\r\n\r\n // Inserts a key only if it's not already present\r\n bool add(K key, const int tid=0) {\r\n return tinystm::updateTx([&] () {\r\n return innerPut(key,key);\r\n });\r\n }\r\n\r\n // Returns true only if the key was present\r\n bool remove(K key, const int tid=0) {\r\n return tinystm::updateTx([&] () {\r\n V notused;\r\n bool retval = innerGet(key,notused,false);\r\n if (retval) innerRemove(key);\r\n return retval;\r\n });\r\n }\r\n\r\n bool contains(K key, const int tid=0) {\r\n return tinystm::readTx([&] () {\r\n V notused;\r\n return innerGet(key,notused,false);\r\n });\r\n }\r\n\r\n // This is not fully transactionally but it's ok because we use it only on initialization.\r\n // We could make it fully transactionally, but we would have to increase the size of allocation/store logs.\r\n void addAll(K** keys, int size, const int tid=0) {\r\n for (int i = 0; i < size; i++) add(*keys[i], tid);\r\n }\r\n\r\n static std::string className() { return tinystm::TinySTM::className() + \"-RedBlackTree\"; }\r\n\r\n};\r\n\r\n#endif // _TINY_STM_RED_BLACK_BST_H_\r\n" }, { "path": "datastructures/trevor_brown_abtree/Makefile", "content": "GPP = g++\nFLAGS = -std=c++11 -mcx16 -O3 -g\nFLAGS += -DPHYSICAL_PROCESSORS=48 -DMAX_TID_POW2=64\n\nLDFLAGS += -I./common\nLDFLAGS += -I./common/dcss\nLDFLAGS += -I./common/atomic_ops\nLDFLAGS += -I./common/descriptors\nLDFLAGS += -I./common/recordmgr\nLDFLAGS += -I./common/rq\nLDFLAGS += -I./common/rq/snapcollector\nLDFLAGS += -I./ds/brown_ext_abtree_lf\nLDFLAGS += -lpthread\n\nall: minimal_example\n\nminimal_example:\n\t$(GPP) $(FLAGS) -o $@.out $@.cpp $(LDFLAGS)\n" }, { "path": "datastructures/trevor_brown_abtree/TrevorBrownABTree.hpp", "content": "#ifndef _TREVOR_BROWN_AB_TREE_HP_H_\r\n#define _TREVOR_BROWN_AB_TREE_HP_H_\r\n\r\n#include \r\n#include \r\n#include \r\n#include \"common/ThreadRegistry.hpp\"\r\n#include \"ds/brown_ext_abtree_lf/brown_ext_abtree_lf_adapter.h\"\r\n\r\n/*\r\n * This is a wrapper to Trevor Brown's AB-Tree so we can use it in our benchmarks\r\n * TODO: We've enabled Hazard Pointers as memory reclamation\r\n */\r\n\r\ntemplate\r\nclass TrevorBrownABTree {\r\n\r\n static const int NODE_DEGREE = 16;\r\n const int ANY_KEY = 0;\r\n const int NUM_THREADS = 128;\r\n\r\n //ds_adapter>* tree;\r\n ds_adapter* tree;\r\n\r\npublic:\r\n TrevorBrownABTree(int numThreads) {\r\n //tree = new ds_adapter>(NUM_THREADS, ANY_KEY);\r\n tree = new ds_adapter(NUM_THREADS, ANY_KEY);\r\n }\r\n\r\n ~TrevorBrownABTree() {\r\n delete tree;\r\n }\r\n\r\n // Inserts a key only if it's not already present\r\n bool add(K key, const int tid=0) {\r\n int threadID = tl_tcico.tid;\r\n if (threadID == ThreadCheckInCheckOut::NOT_ASSIGNED) {\r\n threadID = ThreadRegistry::getTID();\r\n tree->initThread(threadID);\r\n }\r\n return tree->insert(threadID, key, (void *) 1) != tree->getNoValue();\r\n }\r\n\r\n // Returns true only if the key was present\r\n bool remove(K key, const int tid=0) {\r\n int threadID = tl_tcico.tid;\r\n if (threadID == ThreadCheckInCheckOut::NOT_ASSIGNED) {\r\n threadID = ThreadRegistry::getTID();\r\n tree->initThread(threadID);\r\n }\r\n return tree->erase(threadID, key) != tree->getNoValue();\r\n }\r\n\r\n bool contains(K key, const int tid=0) {\r\n int threadID = tl_tcico.tid;\r\n if (threadID == ThreadCheckInCheckOut::NOT_ASSIGNED) {\r\n threadID = ThreadRegistry::getTID();\r\n tree->initThread(threadID);\r\n }\r\n return tree->contains(threadID, key);\r\n }\r\n\r\n // This is not fully transactionally but it's ok because we use it only on initialization.\r\n // We could make it fully transactionally, but we would have to increase the size of allocation/store logs.\r\n void addAll(K** keys, int size, const int tid=0) {\r\n for (int i = 0; i < size; i++) add(*keys[i], tid);\r\n }\r\n\r\n static std::string className() { return \"TrevorBrown-AB-Tree\"; }\r\n\r\n};\r\n\r\n#endif // _TREVOR_BROWN_AB_TREE_HP_H_\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/generalize-small.h", "content": "/* char_load */\r\n#if defined(AO_HAVE_char_load_acquire) && !defined(AO_HAVE_char_load)\r\n# define AO_char_load(addr) AO_char_load_acquire(addr)\r\n# define AO_HAVE_char_load\r\n#endif\r\n\r\n#if defined(AO_HAVE_char_load_full) && !defined(AO_HAVE_char_load_acquire)\r\n# define AO_char_load_acquire(addr) AO_char_load_full(addr)\r\n# define AO_HAVE_char_load_acquire\r\n#endif\r\n\r\n#if defined(AO_HAVE_char_load_full) && !defined(AO_HAVE_char_load_read)\r\n# define AO_char_load_read(addr) AO_char_load_full(addr)\r\n# define AO_HAVE_char_load_read\r\n#endif\r\n\r\n#if !defined(AO_HAVE_char_load_acquire_read) && defined(AO_HAVE_char_load_acquire)\r\n# define AO_char_load_acquire_read(addr) AO_char_load_acquire(addr)\r\n# define AO_HAVE_char_load_acquire_read\r\n#endif\r\n\r\n#if defined(AO_HAVE_char_load) && defined(AO_HAVE_nop_full) && \\\r\n !defined(AO_HAVE_char_load_acquire)\r\n AO_INLINE unsigned char\r\n AO_char_load_acquire(const volatile unsigned char *addr)\r\n {\r\n unsigned char result = AO_char_load(addr);\r\n /* Acquire barrier would be useless, since the load could be delayed */\r\n /* beyond it. */\r\n AO_nop_full();\r\n return result;\r\n }\r\n# define AO_HAVE_char_load_acquire\r\n#endif\r\n\r\n#if defined(AO_HAVE_char_load) && defined(AO_HAVE_nop_read) && \\\r\n !defined(AO_HAVE_char_load_read)\r\n AO_INLINE unsigned char\r\n AO_char_load_read(const volatile unsigned char *addr)\r\n {\r\n unsigned char result = AO_char_load(addr);\r\n /* Acquire barrier would be useless, since the load could be delayed */\r\n /* beyond it. */\r\n AO_nop_read();\r\n return result;\r\n }\r\n# define AO_HAVE_char_load_read\r\n#endif\r\n\r\n#if defined(AO_HAVE_char_load_acquire) && defined(AO_HAVE_nop_full) && \\\r\n !defined(AO_HAVE_char_load_full)\r\n# define AO_char_load_full(addr) (AO_nop_full(), AO_char_load_acquire(addr))\r\n# define AO_HAVE_char_load_full\r\n#endif\r\n\r\n#if !defined(AO_HAVE_char_load_acquire_read) && defined(AO_HAVE_char_load_read)\r\n# define AO_char_load_acquire_read(addr) AO_char_load_read(addr)\r\n# define AO_HAVE_char_load_acquire_read\r\n#endif\r\n\r\n#if defined(AO_HAVE_char_load_acquire_read) && !defined(AO_HAVE_char_load)\r\n# define AO_char_load(addr) AO_char_load_acquire_read(addr)\r\n# define AO_HAVE_char_load\r\n#endif\r\n\r\n#ifdef AO_NO_DD_ORDERING\r\n# if defined(AO_HAVE_char_load_acquire_read)\r\n# define AO_char_load_dd_acquire_read(addr) \\\r\n AO_char_load_acquire_read(addr)\r\n# define AO_HAVE_char_load_dd_acquire_read\r\n# endif\r\n#else\r\n# if defined(AO_HAVE_char_load)\r\n# define AO_char_load_dd_acquire_read(addr) \\\r\n AO_char_load(addr)\r\n# define AO_HAVE_char_load_dd_acquire_read\r\n# endif\r\n#endif\r\n\r\n\r\n/* char_store */\r\n\r\n#if defined(AO_HAVE_char_store_release) && !defined(AO_HAVE_char_store)\r\n# define AO_char_store(addr, val) AO_char_store_release(addr,val)\r\n# define AO_HAVE_char_store\r\n#endif\r\n\r\n#if defined(AO_HAVE_char_store_full) && !defined(AO_HAVE_char_store_release)\r\n# define AO_char_store_release(addr,val) AO_char_store_full(addr,val)\r\n# define AO_HAVE_char_store_release\r\n#endif\r\n\r\n#if defined(AO_HAVE_char_store_full) && !defined(AO_HAVE_char_store_write)\r\n# define AO_char_store_write(addr,val) AO_char_store_full(addr,val)\r\n# define AO_HAVE_char_store_write\r\n#endif\r\n\r\n#if defined(AO_HAVE_char_store_release) && \\\r\n !defined(AO_HAVE_char_store_release_write)\r\n# define AO_char_store_release_write(addr, val) \\\r\n AO_char_store_release(addr,val)\r\n# define AO_HAVE_char_store_release_write\r\n#endif\r\n\r\n#if defined(AO_HAVE_char_store_write) && !defined(AO_HAVE_char_store)\r\n# define AO_char_store(addr, val) AO_char_store_write(addr,val)\r\n# define AO_HAVE_char_store\r\n#endif\r\n\r\n#if defined(AO_HAVE_char_store) && defined(AO_HAVE_nop_full) && \\\r\n !defined(AO_HAVE_char_store_release)\r\n# define AO_char_store_release(addr,val) \\\r\n (AO_nop_full(), AO_char_store(addr,val))\r\n# define AO_HAVE_char_store_release\r\n#endif\r\n\r\n#if defined(AO_HAVE_nop_write) && defined(AO_HAVE_char_store) && \\\r\n !defined(AO_HAVE_char_store_write)\r\n# define AO_char_store_write(addr, val) \\\r\n (AO_nop_write(), AO_char_store(addr,val))\r\n# define AO_HAVE_char_store_write\r\n#endif\r\n\r\n#if defined(AO_HAVE_char_store_write) && \\\r\n !defined(AO_HAVE_char_store_release_write)\r\n# define AO_char_store_release_write(addr, val) AO_char_store_write(addr,val)\r\n# define AO_HAVE_char_store_release_write\r\n#endif\r\n\r\n#if defined(AO_HAVE_char_store_release) && defined(AO_HAVE_nop_full) && \\\r\n !defined(AO_HAVE_char_store_full)\r\n# define AO_char_store_full(addr, val) \\\r\n (AO_char_store_release(addr, val), AO_nop_full())\r\n# define AO_HAVE_char_store_full\r\n#endif\r\n\r\n\r\n/* char_fetch_and_add */\r\n#if defined(AO_HAVE_char_compare_and_swap_full) && \\\r\n !defined(AO_HAVE_char_fetch_and_add_full)\r\n AO_INLINE AO_t\r\n AO_char_fetch_and_add_full(volatile unsigned char *addr,\r\n unsigned char incr)\r\n {\r\n unsigned char old;\r\n do\r\n {\r\n old = *addr;\r\n }\r\n while (!AO_char_compare_and_swap_full(addr, old, old+incr));\r\n return old;\r\n }\r\n# define AO_HAVE_char_fetch_and_add_full\r\n#endif\r\n\r\n#if defined(AO_HAVE_char_compare_and_swap_acquire) && \\\r\n !defined(AO_HAVE_char_fetch_and_add_acquire)\r\n AO_INLINE AO_t\r\n AO_char_fetch_and_add_acquire(volatile unsigned char *addr,\r\n unsigned char incr)\r\n {\r\n unsigned char old;\r\n do\r\n {\r\n old = *addr;\r\n }\r\n while (!AO_char_compare_and_swap_acquire(addr, old, old+incr));\r\n return old;\r\n }\r\n# define AO_HAVE_char_fetch_and_add_acquire\r\n#endif\r\n\r\n#if defined(AO_HAVE_char_compare_and_swap_release) && \\\r\n !defined(AO_HAVE_char_fetch_and_add_release)\r\n AO_INLINE AO_t\r\n AO_char_fetch_and_add_release(volatile unsigned char *addr,\r\n unsigned char incr)\r\n {\r\n unsigned char old;\r\n do\r\n {\r\n old = *addr;\r\n }\r\n while (!AO_char_compare_and_swap_release(addr, old, old+incr));\r\n return old;\r\n }\r\n# define AO_HAVE_char_fetch_and_add_release\r\n#endif\r\n\r\n#if defined(AO_HAVE_char_fetch_and_add_full)\r\n# if !defined(AO_HAVE_char_fetch_and_add_release)\r\n# define AO_char_fetch_and_add_release(addr, val) \\\r\n AO_char_fetch_and_add_full(addr, val)\r\n# define AO_HAVE_char_fetch_and_add_release\r\n# endif\r\n# if !defined(AO_HAVE_char_fetch_and_add_acquire)\r\n# define AO_char_fetch_and_add_acquire(addr, val) \\\r\n AO_char_fetch_and_add_full(addr, val)\r\n# define AO_HAVE_char_fetch_and_add_acquire\r\n# endif\r\n# if !defined(AO_HAVE_char_fetch_and_add_write)\r\n# define AO_char_fetch_and_add_write(addr, val) \\\r\n AO_char_fetch_and_add_full(addr, val)\r\n# define AO_HAVE_char_fetch_and_add_write\r\n# endif\r\n# if !defined(AO_HAVE_char_fetch_and_add_read)\r\n# define AO_char_fetch_and_add_read(addr, val) \\\r\n AO_char_fetch_and_add_full(addr, val)\r\n# define AO_HAVE_char_fetch_and_add_read\r\n# endif\r\n#endif /* AO_HAVE_char_fetch_and_add_full */\r\n\r\n#if !defined(AO_HAVE_char_fetch_and_add) && \\\r\n defined(AO_HAVE_char_fetch_and_add_release)\r\n# define AO_char_fetch_and_add(addr, val) \\\r\n AO_char_fetch_and_add_release(addr, val)\r\n# define AO_HAVE_char_fetch_and_add\r\n#endif\r\n#if !defined(AO_HAVE_char_fetch_and_add) && \\\r\n defined(AO_HAVE_char_fetch_and_add_acquire)\r\n# define AO_char_fetch_and_add(addr, val) \\\r\n AO_char_fetch_and_add_acquire(addr, val)\r\n# define AO_HAVE_char_fetch_and_add\r\n#endif\r\n#if !defined(AO_HAVE_char_fetch_and_add) && \\\r\n defined(AO_HAVE_char_fetch_and_add_write)\r\n# define AO_char_fetch_and_add(addr, val) \\\r\n AO_char_fetch_and_add_write(addr, val)\r\n# define AO_HAVE_char_fetch_and_add\r\n#endif\r\n#if !defined(AO_HAVE_char_fetch_and_add) && \\\r\n defined(AO_HAVE_char_fetch_and_add_read)\r\n# define AO_char_fetch_and_add(addr, val) \\\r\n AO_char_fetch_and_add_read(addr, val)\r\n# define AO_HAVE_char_fetch_and_add\r\n#endif\r\n\r\n#if defined(AO_HAVE_char_fetch_and_add_acquire) &&\\\r\n defined(AO_HAVE_nop_full) && \\\r\n !defined(AO_HAVE_char_fetch_and_add_full)\r\n# define AO_char_fetch_and_add_full(addr, val) \\\r\n (AO_nop_full(), AO_char_fetch_and_add_acquire(addr, val))\r\n#endif\r\n\r\n#if !defined(AO_HAVE_char_fetch_and_add_release_write) && \\\r\n defined(AO_HAVE_char_fetch_and_add_write)\r\n# define AO_char_fetch_and_add_release_write(addr, val) \\\r\n AO_char_fetch_and_add_write(addr, val)\r\n# define AO_HAVE_char_fetch_and_add_release_write\r\n#endif\r\n#if !defined(AO_HAVE_char_fetch_and_add_release_write) && \\\r\n defined(AO_HAVE_char_fetch_and_add_release)\r\n# define AO_char_fetch_and_add_release_write(addr, val) \\\r\n AO_char_fetch_and_add_release(addr, val)\r\n# define AO_HAVE_char_fetch_and_add_release_write\r\n#endif\r\n#if !defined(AO_HAVE_char_fetch_and_add_acquire_read) && \\\r\n defined(AO_HAVE_char_fetch_and_add_read)\r\n# define AO_char_fetch_and_add_acquire_read(addr, val) \\\r\n AO_char_fetch_and_add_read(addr, val)\r\n# define AO_HAVE_char_fetch_and_add_acquire_read\r\n#endif\r\n#if !defined(AO_HAVE_char_fetch_and_add_acquire_read) && \\\r\n defined(AO_HAVE_char_fetch_and_add_acquire)\r\n# define AO_char_fetch_and_add_acquire_read(addr, val) \\\r\n AO_char_fetch_and_add_acquire(addr, val)\r\n# define AO_HAVE_char_fetch_and_add_acquire_read\r\n#endif\r\n\r\n#ifdef AO_NO_DD_ORDERING\r\n# if defined(AO_HAVE_char_fetch_and_add_acquire_read)\r\n# define AO_char_fetch_and_add_dd_acquire_read(addr, val) \\\r\n AO_char_fetch_and_add_acquire_read(addr, val)\r\n# define AO_HAVE_char_fetch_and_add_dd_acquire_read\r\n# endif\r\n#else\r\n# if defined(AO_HAVE_char_fetch_and_add)\r\n# define AO_char_fetch_and_add_dd_acquire_read(addr, val) \\\r\n AO_char_fetch_and_add(addr, val)\r\n# define AO_HAVE_char_fetch_and_add_dd_acquire_read\r\n# endif\r\n#endif\r\n\r\n/* char_fetch_and_add1 */\r\n\r\n#if defined(AO_HAVE_char_fetch_and_add_full) &&\\\r\n !defined(AO_HAVE_char_fetch_and_add1_full)\r\n# define AO_char_fetch_and_add1_full(addr) \\\r\n AO_char_fetch_and_add_full(addr,1)\r\n# define AO_HAVE_char_fetch_and_add1_full\r\n#endif\r\n#if defined(AO_HAVE_char_fetch_and_add_release) &&\\\r\n !defined(AO_HAVE_char_fetch_and_add1_release)\r\n# define AO_char_fetch_and_add1_release(addr) \\\r\n AO_char_fetch_and_add_release(addr,1)\r\n# define AO_HAVE_char_fetch_and_add1_release\r\n#endif\r\n#if defined(AO_HAVE_char_fetch_and_add_acquire) &&\\\r\n !defined(AO_HAVE_char_fetch_and_add1_acquire)\r\n# define AO_char_fetch_and_add1_acquire(addr) \\\r\n AO_char_fetch_and_add_acquire(addr,1)\r\n# define AO_HAVE_char_fetch_and_add1_acquire\r\n#endif\r\n#if defined(AO_HAVE_char_fetch_and_add_write) &&\\\r\n !defined(AO_HAVE_char_fetch_and_add1_write)\r\n# define AO_char_fetch_and_add1_write(addr) \\\r\n AO_char_fetch_and_add_write(addr,1)\r\n# define AO_HAVE_char_fetch_and_add1_write\r\n#endif\r\n#if defined(AO_HAVE_char_fetch_and_add_read) &&\\\r\n !defined(AO_HAVE_char_fetch_and_add1_read)\r\n# define AO_char_fetch_and_add1_read(addr) \\\r\n AO_char_fetch_and_add_read(addr,1)\r\n# define AO_HAVE_char_fetch_and_add1_read\r\n#endif\r\n#if defined(AO_HAVE_char_fetch_and_add_release_write) &&\\\r\n !defined(AO_HAVE_char_fetch_and_add1_release_write)\r\n# define AO_char_fetch_and_add1_release_write(addr) \\\r\n AO_char_fetch_and_add_release_write(addr,1)\r\n# define AO_HAVE_char_fetch_and_add1_release_write\r\n#endif\r\n#if defined(AO_HAVE_char_fetch_and_add_acquire_read) &&\\\r\n !defined(AO_HAVE_char_fetch_and_add1_acquire_read)\r\n# define AO_char_fetch_and_add1_acquire_read(addr) \\\r\n AO_char_fetch_and_add_acquire_read(addr,1)\r\n# define AO_HAVE_char_fetch_and_add1_acquire_read\r\n#endif\r\n#if defined(AO_HAVE_char_fetch_and_add) &&\\\r\n !defined(AO_HAVE_char_fetch_and_add1)\r\n# define AO_char_fetch_and_add1(addr) \\\r\n AO_char_fetch_and_add(addr,1)\r\n# define AO_HAVE_char_fetch_and_add1\r\n#endif\r\n\r\n#if defined(AO_HAVE_char_fetch_and_add1_full)\r\n# if !defined(AO_HAVE_char_fetch_and_add1_release)\r\n# define AO_char_fetch_and_add1_release(addr) \\\r\n AO_char_fetch_and_add1_full(addr)\r\n# define AO_HAVE_char_fetch_and_add1_release\r\n# endif\r\n# if !defined(AO_HAVE_char_fetch_and_add1_acquire)\r\n# define AO_char_fetch_and_add1_acquire(addr) \\\r\n AO_char_fetch_and_add1_full(addr)\r\n# define AO_HAVE_char_fetch_and_add1_acquire\r\n# endif\r\n# if !defined(AO_HAVE_char_fetch_and_add1_write)\r\n# define AO_char_fetch_and_add1_write(addr) \\\r\n AO_char_fetch_and_add1_full(addr)\r\n# define AO_HAVE_char_fetch_and_add1_write\r\n# endif\r\n# if !defined(AO_HAVE_char_fetch_and_add1_read)\r\n# define AO_char_fetch_and_add1_read(addr) \\\r\n AO_char_fetch_and_add1_full(addr)\r\n# define AO_HAVE_char_fetch_and_add1_read\r\n# endif\r\n#endif /* AO_HAVE_char_fetch_and_add1_full */\r\n\r\n#if !defined(AO_HAVE_char_fetch_and_add1) && \\\r\n defined(AO_HAVE_char_fetch_and_add1_release)\r\n# define AO_char_fetch_and_add1(addr) \\\r\n AO_char_fetch_and_add1_release(addr)\r\n# define AO_HAVE_char_fetch_and_add1\r\n#endif\r\n#if !defined(AO_HAVE_char_fetch_and_add1) && \\\r\n defined(AO_HAVE_char_fetch_and_add1_acquire)\r\n# define AO_char_fetch_and_add1(addr) \\\r\n AO_char_fetch_and_add1_acquire(addr)\r\n# define AO_HAVE_char_fetch_and_add1\r\n#endif\r\n#if !defined(AO_HAVE_char_fetch_and_add1) && \\\r\n defined(AO_HAVE_char_fetch_and_add1_write)\r\n# define AO_char_fetch_and_add1(addr) \\\r\n AO_char_fetch_and_add1_write(addr)\r\n# define AO_HAVE_char_fetch_and_add1\r\n#endif\r\n#if !defined(AO_HAVE_char_fetch_and_add1) && \\\r\n defined(AO_HAVE_char_fetch_and_add1_read)\r\n# define AO_char_fetch_and_add1(addr) \\\r\n AO_char_fetch_and_add1_read(addr)\r\n# define AO_HAVE_char_fetch_and_add1\r\n#endif\r\n\r\n#if defined(AO_HAVE_char_fetch_and_add1_acquire) &&\\\r\n defined(AO_HAVE_nop_full) && \\\r\n !defined(AO_HAVE_char_fetch_and_add1_full)\r\n# define AO_char_fetch_and_add1_full(addr) \\\r\n (AO_nop_full(), AO_char_fetch_and_add1_acquire(addr))\r\n# define AO_HAVE_char_fetch_and_add1_full\r\n#endif\r\n\r\n#if !defined(AO_HAVE_char_fetch_and_add1_release_write) && \\\r\n defined(AO_HAVE_char_fetch_and_add1_write)\r\n# define AO_char_fetch_and_add1_release_write(addr) \\\r\n AO_char_fetch_and_add1_write(addr)\r\n# define AO_HAVE_char_fetch_and_add1_release_write\r\n#endif\r\n#if !defined(AO_HAVE_char_fetch_and_add1_release_write) && \\\r\n defined(AO_HAVE_char_fetch_and_add1_release)\r\n# define AO_char_fetch_and_add1_release_write(addr) \\\r\n AO_char_fetch_and_add1_release(addr)\r\n# define AO_HAVE_char_fetch_and_add1_release_write\r\n#endif\r\n#if !defined(AO_HAVE_char_fetch_and_add1_acquire_read) && \\\r\n defined(AO_HAVE_char_fetch_and_add1_read)\r\n# define AO_char_fetch_and_add1_acquire_read(addr) \\\r\n AO_char_fetch_and_add1_read(addr)\r\n# define AO_HAVE_char_fetch_and_add1_acquire_read\r\n#endif\r\n#if !defined(AO_HAVE_char_fetch_and_add1_acquire_read) && \\\r\n defined(AO_HAVE_char_fetch_and_add1_acquire)\r\n# define AO_char_fetch_and_add1_acquire_read(addr) \\\r\n AO_char_fetch_and_add1_acquire(addr)\r\n# define AO_HAVE_char_fetch_and_add1_acquire_read\r\n#endif\r\n\r\n#ifdef AO_NO_DD_ORDERING\r\n# if defined(AO_HAVE_char_fetch_and_add1_acquire_read)\r\n# define AO_char_fetch_and_add1_dd_acquire_read(addr) \\\r\n AO_char_fetch_and_add1_acquire_read(addr)\r\n# define AO_HAVE_char_fetch_and_add1_dd_acquire_read\r\n# endif\r\n#else\r\n# if defined(AO_HAVE_char_fetch_and_add1)\r\n# define AO_char_fetch_and_add1_dd_acquire_read(addr) \\\r\n AO_char_fetch_and_add1(addr)\r\n# define AO_HAVE_char_fetch_and_add1_dd_acquire_read\r\n# endif\r\n#endif\r\n\r\n/* char_fetch_and_sub1 */\r\n\r\n#if defined(AO_HAVE_char_fetch_and_add_full) &&\\\r\n !defined(AO_HAVE_char_fetch_and_sub1_full)\r\n# define AO_char_fetch_and_sub1_full(addr) \\\r\n AO_char_fetch_and_add_full(addr,(unsigned char)(-1))\r\n# define AO_HAVE_char_fetch_and_sub1_full\r\n#endif\r\n#if defined(AO_HAVE_char_fetch_and_add_release) &&\\\r\n !defined(AO_HAVE_char_fetch_and_sub1_release)\r\n# define AO_char_fetch_and_sub1_release(addr) \\\r\n AO_char_fetch_and_add_release(addr,(unsigned char)(-1))\r\n# define AO_HAVE_char_fetch_and_sub1_release\r\n#endif\r\n#if defined(AO_HAVE_char_fetch_and_add_acquire) &&\\\r\n !defined(AO_HAVE_char_fetch_and_sub1_acquire)\r\n# define AO_char_fetch_and_sub1_acquire(addr) \\\r\n AO_char_fetch_and_add_acquire(addr,(unsigned char)(-1))\r\n# define AO_HAVE_char_fetch_and_sub1_acquire\r\n#endif\r\n#if defined(AO_HAVE_char_fetch_and_add_write) &&\\\r\n !defined(AO_HAVE_char_fetch_and_sub1_write)\r\n# define AO_char_fetch_and_sub1_write(addr) \\\r\n AO_char_fetch_and_add_write(addr,(unsigned char)(-1))\r\n# define AO_HAVE_char_fetch_and_sub1_write\r\n#endif\r\n#if defined(AO_HAVE_char_fetch_and_add_read) &&\\\r\n !defined(AO_HAVE_char_fetch_and_sub1_read)\r\n# define AO_char_fetch_and_sub1_read(addr) \\\r\n AO_char_fetch_and_add_read(addr,(unsigned char)(-1))\r\n# define AO_HAVE_char_fetch_and_sub1_read\r\n#endif\r\n#if defined(AO_HAVE_char_fetch_and_add_release_write) &&\\\r\n !defined(AO_HAVE_char_fetch_and_sub1_release_write)\r\n# define AO_char_fetch_and_sub1_release_write(addr) \\\r\n AO_char_fetch_and_add_release_write(addr,(unsigned char)(-1))\r\n# define AO_HAVE_char_fetch_and_sub1_release_write\r\n#endif\r\n#if defined(AO_HAVE_char_fetch_and_add_acquire_read) &&\\\r\n !defined(AO_HAVE_char_fetch_and_sub1_acquire_read)\r\n# define AO_char_fetch_and_sub1_acquire_read(addr) \\\r\n AO_char_fetch_and_add_acquire_read(addr,(unsigned char)(-1))\r\n# define AO_HAVE_char_fetch_and_sub1_acquire_read\r\n#endif\r\n#if defined(AO_HAVE_char_fetch_and_add) &&\\\r\n !defined(AO_HAVE_char_fetch_and_sub1)\r\n# define AO_char_fetch_and_sub1(addr) \\\r\n AO_char_fetch_and_add(addr,(unsigned char)(-1))\r\n# define AO_HAVE_char_fetch_and_sub1\r\n#endif\r\n\r\n#if defined(AO_HAVE_char_fetch_and_sub1_full)\r\n# if !defined(AO_HAVE_char_fetch_and_sub1_release)\r\n# define AO_char_fetch_and_sub1_release(addr) \\\r\n AO_char_fetch_and_sub1_full(addr)\r\n# define AO_HAVE_char_fetch_and_sub1_release\r\n# endif\r\n# if !defined(AO_HAVE_char_fetch_and_sub1_acquire)\r\n# define AO_char_fetch_and_sub1_acquire(addr) \\\r\n AO_char_fetch_and_sub1_full(addr)\r\n# define AO_HAVE_char_fetch_and_sub1_acquire\r\n# endif\r\n# if !defined(AO_HAVE_char_fetch_and_sub1_write)\r\n# define AO_char_fetch_and_sub1_write(addr) \\\r\n AO_char_fetch_and_sub1_full(addr)\r\n# define AO_HAVE_char_fetch_and_sub1_write\r\n# endif\r\n# if !defined(AO_HAVE_char_fetch_and_sub1_read)\r\n# define AO_char_fetch_and_sub1_read(addr) \\\r\n AO_char_fetch_and_sub1_full(addr)\r\n# define AO_HAVE_char_fetch_and_sub1_read\r\n# endif\r\n#endif /* AO_HAVE_char_fetch_and_sub1_full */\r\n\r\n#if !defined(AO_HAVE_char_fetch_and_sub1) && \\\r\n defined(AO_HAVE_char_fetch_and_sub1_release)\r\n# define AO_char_fetch_and_sub1(addr) \\\r\n AO_char_fetch_and_sub1_release(addr)\r\n# define AO_HAVE_char_fetch_and_sub1\r\n#endif\r\n#if !defined(AO_HAVE_char_fetch_and_sub1) && \\\r\n defined(AO_HAVE_char_fetch_and_sub1_acquire)\r\n# define AO_char_fetch_and_sub1(addr) \\\r\n AO_char_fetch_and_sub1_acquire(addr)\r\n# define AO_HAVE_char_fetch_and_sub1\r\n#endif\r\n#if !defined(AO_HAVE_char_fetch_and_sub1) && \\\r\n defined(AO_HAVE_char_fetch_and_sub1_write)\r\n# define AO_char_fetch_and_sub1(addr) \\\r\n AO_char_fetch_and_sub1_write(addr)\r\n# define AO_HAVE_char_fetch_and_sub1\r\n#endif\r\n#if !defined(AO_HAVE_char_fetch_and_sub1) && \\\r\n defined(AO_HAVE_char_fetch_and_sub1_read)\r\n# define AO_char_fetch_and_sub1(addr) \\\r\n AO_char_fetch_and_sub1_read(addr)\r\n# define AO_HAVE_char_fetch_and_sub1\r\n#endif\r\n\r\n#if defined(AO_HAVE_char_fetch_and_sub1_acquire) &&\\\r\n defined(AO_HAVE_nop_full) && \\\r\n !defined(AO_HAVE_char_fetch_and_sub1_full)\r\n# define AO_char_fetch_and_sub1_full(addr) \\\r\n (AO_nop_full(), AO_char_fetch_and_sub1_acquire(addr))\r\n# define AO_HAVE_char_fetch_and_sub1_full\r\n#endif\r\n\r\n#if !defined(AO_HAVE_char_fetch_and_sub1_release_write) && \\\r\n defined(AO_HAVE_char_fetch_and_sub1_write)\r\n# define AO_char_fetch_and_sub1_release_write(addr) \\\r\n AO_char_fetch_and_sub1_write(addr)\r\n# define AO_HAVE_char_fetch_and_sub1_release_write\r\n#endif\r\n#if !defined(AO_HAVE_char_fetch_and_sub1_release_write) && \\\r\n defined(AO_HAVE_char_fetch_and_sub1_release)\r\n# define AO_char_fetch_and_sub1_release_write(addr) \\\r\n AO_char_fetch_and_sub1_release(addr)\r\n# define AO_HAVE_char_fetch_and_sub1_release_write\r\n#endif\r\n#if !defined(AO_HAVE_char_fetch_and_sub1_acquire_read) && \\\r\n defined(AO_HAVE_char_fetch_and_sub1_read)\r\n# define AO_char_fetch_and_sub1_acquire_read(addr) \\\r\n AO_char_fetch_and_sub1_read(addr)\r\n# define AO_HAVE_char_fetch_and_sub1_acquire_read\r\n#endif\r\n#if !defined(AO_HAVE_char_fetch_and_sub1_acquire_read) && \\\r\n defined(AO_HAVE_char_fetch_and_sub1_acquire)\r\n# define AO_char_fetch_and_sub1_acquire_read(addr) \\\r\n AO_char_fetch_and_sub1_acquire(addr)\r\n# define AO_HAVE_char_fetch_and_sub1_acquire_read\r\n#endif\r\n\r\n#ifdef AO_NO_DD_ORDERING\r\n# if defined(AO_HAVE_char_fetch_and_sub1_acquire_read)\r\n# define AO_char_fetch_and_sub1_dd_acquire_read(addr) \\\r\n AO_char_fetch_and_sub1_acquire_read(addr)\r\n# define AO_HAVE_char_fetch_and_sub1_dd_acquire_read\r\n# endif\r\n#else\r\n# if defined(AO_HAVE_char_fetch_and_sub1)\r\n# define AO_char_fetch_and_sub1_dd_acquire_read(addr) \\\r\n AO_char_fetch_and_sub1(addr)\r\n# define AO_HAVE_char_fetch_and_sub1_dd_acquire_read\r\n# endif\r\n#endif\r\n\r\n/* short_load */\r\n#if defined(AO_HAVE_short_load_acquire) && !defined(AO_HAVE_short_load)\r\n# define AO_short_load(addr) AO_short_load_acquire(addr)\r\n# define AO_HAVE_short_load\r\n#endif\r\n\r\n#if defined(AO_HAVE_short_load_full) && !defined(AO_HAVE_short_load_acquire)\r\n# define AO_short_load_acquire(addr) AO_short_load_full(addr)\r\n# define AO_HAVE_short_load_acquire\r\n#endif\r\n\r\n#if defined(AO_HAVE_short_load_full) && !defined(AO_HAVE_short_load_read)\r\n# define AO_short_load_read(addr) AO_short_load_full(addr)\r\n# define AO_HAVE_short_load_read\r\n#endif\r\n\r\n#if !defined(AO_HAVE_short_load_acquire_read) && defined(AO_HAVE_short_load_acquire)\r\n# define AO_short_load_acquire_read(addr) AO_short_load_acquire(addr)\r\n# define AO_HAVE_short_load_acquire_read\r\n#endif\r\n\r\n#if defined(AO_HAVE_short_load) && defined(AO_HAVE_nop_full) && \\\r\n !defined(AO_HAVE_short_load_acquire)\r\n AO_INLINE unsigned short\r\n AO_short_load_acquire(const volatile unsigned short *addr)\r\n {\r\n unsigned short result = AO_short_load(addr);\r\n /* Acquire barrier would be useless, since the load could be delayed */\r\n /* beyond it. */\r\n AO_nop_full();\r\n return result;\r\n }\r\n# define AO_HAVE_short_load_acquire\r\n#endif\r\n\r\n#if defined(AO_HAVE_short_load) && defined(AO_HAVE_nop_read) && \\\r\n !defined(AO_HAVE_short_load_read)\r\n AO_INLINE unsigned short\r\n AO_short_load_read(const volatile unsigned short *addr)\r\n {\r\n unsigned short result = AO_short_load(addr);\r\n /* Acquire barrier would be useless, since the load could be delayed */\r\n /* beyond it. */\r\n AO_nop_read();\r\n return result;\r\n }\r\n# define AO_HAVE_short_load_read\r\n#endif\r\n\r\n#if defined(AO_HAVE_short_load_acquire) && defined(AO_HAVE_nop_full) && \\\r\n !defined(AO_HAVE_short_load_full)\r\n# define AO_short_load_full(addr) (AO_nop_full(), AO_short_load_acquire(addr))\r\n# define AO_HAVE_short_load_full\r\n#endif\r\n\r\n#if !defined(AO_HAVE_short_load_acquire_read) && defined(AO_HAVE_short_load_read)\r\n# define AO_short_load_acquire_read(addr) AO_short_load_read(addr)\r\n# define AO_HAVE_short_load_acquire_read\r\n#endif\r\n\r\n#if defined(AO_HAVE_short_load_acquire_read) && !defined(AO_HAVE_short_load)\r\n# define AO_short_load(addr) AO_short_load_acquire_read(addr)\r\n# define AO_HAVE_short_load\r\n#endif\r\n\r\n#ifdef AO_NO_DD_ORDERING\r\n# if defined(AO_HAVE_short_load_acquire_read)\r\n# define AO_short_load_dd_acquire_read(addr) \\\r\n AO_short_load_acquire_read(addr)\r\n# define AO_HAVE_short_load_dd_acquire_read\r\n# endif\r\n#else\r\n# if defined(AO_HAVE_short_load)\r\n# define AO_short_load_dd_acquire_read(addr) \\\r\n AO_short_load(addr)\r\n# define AO_HAVE_short_load_dd_acquire_read\r\n# endif\r\n#endif\r\n\r\n\r\n/* short_store */\r\n\r\n#if defined(AO_HAVE_short_store_release) && !defined(AO_HAVE_short_store)\r\n# define AO_short_store(addr, val) AO_short_store_release(addr,val)\r\n# define AO_HAVE_short_store\r\n#endif\r\n\r\n#if defined(AO_HAVE_short_store_full) && !defined(AO_HAVE_short_store_release)\r\n# define AO_short_store_release(addr,val) AO_short_store_full(addr,val)\r\n# define AO_HAVE_short_store_release\r\n#endif\r\n\r\n#if defined(AO_HAVE_short_store_full) && !defined(AO_HAVE_short_store_write)\r\n# define AO_short_store_write(addr,val) AO_short_store_full(addr,val)\r\n# define AO_HAVE_short_store_write\r\n#endif\r\n\r\n#if defined(AO_HAVE_short_store_release) && \\\r\n !defined(AO_HAVE_short_store_release_write)\r\n# define AO_short_store_release_write(addr, val) \\\r\n AO_short_store_release(addr,val)\r\n# define AO_HAVE_short_store_release_write\r\n#endif\r\n\r\n#if defined(AO_HAVE_short_store_write) && !defined(AO_HAVE_short_store)\r\n# define AO_short_store(addr, val) AO_short_store_write(addr,val)\r\n# define AO_HAVE_short_store\r\n#endif\r\n\r\n#if defined(AO_HAVE_short_store) && defined(AO_HAVE_nop_full) && \\\r\n !defined(AO_HAVE_short_store_release)\r\n# define AO_short_store_release(addr,val) \\\r\n (AO_nop_full(), AO_short_store(addr,val))\r\n# define AO_HAVE_short_store_release\r\n#endif\r\n\r\n#if defined(AO_HAVE_nop_write) && defined(AO_HAVE_short_store) && \\\r\n !defined(AO_HAVE_short_store_write)\r\n# define AO_short_store_write(addr, val) \\\r\n (AO_nop_write(), AO_short_store(addr,val))\r\n# define AO_HAVE_short_store_write\r\n#endif\r\n\r\n#if defined(AO_HAVE_short_store_write) && \\\r\n !defined(AO_HAVE_short_store_release_write)\r\n# define AO_short_store_release_write(addr, val) AO_short_store_write(addr,val)\r\n# define AO_HAVE_short_store_release_write\r\n#endif\r\n\r\n#if defined(AO_HAVE_short_store_release) && defined(AO_HAVE_nop_full) && \\\r\n !defined(AO_HAVE_short_store_full)\r\n# define AO_short_store_full(addr, val) \\\r\n (AO_short_store_release(addr, val), AO_nop_full())\r\n# define AO_HAVE_short_store_full\r\n#endif\r\n\r\n\r\n/* short_fetch_and_add */\r\n#if defined(AO_HAVE_short_compare_and_swap_full) && \\\r\n !defined(AO_HAVE_short_fetch_and_add_full)\r\n AO_INLINE AO_t\r\n AO_short_fetch_and_add_full(volatile unsigned short *addr,\r\n unsigned short incr)\r\n {\r\n unsigned short old;\r\n do\r\n {\r\n old = *addr;\r\n }\r\n while (!AO_short_compare_and_swap_full(addr, old, old+incr));\r\n return old;\r\n }\r\n# define AO_HAVE_short_fetch_and_add_full\r\n#endif\r\n\r\n#if defined(AO_HAVE_short_compare_and_swap_acquire) && \\\r\n !defined(AO_HAVE_short_fetch_and_add_acquire)\r\n AO_INLINE AO_t\r\n AO_short_fetch_and_add_acquire(volatile unsigned short *addr,\r\n unsigned short incr)\r\n {\r\n unsigned short old;\r\n do\r\n {\r\n old = *addr;\r\n }\r\n while (!AO_short_compare_and_swap_acquire(addr, old, old+incr));\r\n return old;\r\n }\r\n# define AO_HAVE_short_fetch_and_add_acquire\r\n#endif\r\n\r\n#if defined(AO_HAVE_short_compare_and_swap_release) && \\\r\n !defined(AO_HAVE_short_fetch_and_add_release)\r\n AO_INLINE AO_t\r\n AO_short_fetch_and_add_release(volatile unsigned short *addr,\r\n unsigned short incr)\r\n {\r\n unsigned short old;\r\n do\r\n {\r\n old = *addr;\r\n }\r\n while (!AO_short_compare_and_swap_release(addr, old, old+incr));\r\n return old;\r\n }\r\n# define AO_HAVE_short_fetch_and_add_release\r\n#endif\r\n\r\n#if defined(AO_HAVE_short_fetch_and_add_full)\r\n# if !defined(AO_HAVE_short_fetch_and_add_release)\r\n# define AO_short_fetch_and_add_release(addr, val) \\\r\n AO_short_fetch_and_add_full(addr, val)\r\n# define AO_HAVE_short_fetch_and_add_release\r\n# endif\r\n# if !defined(AO_HAVE_short_fetch_and_add_acquire)\r\n# define AO_short_fetch_and_add_acquire(addr, val) \\\r\n AO_short_fetch_and_add_full(addr, val)\r\n# define AO_HAVE_short_fetch_and_add_acquire\r\n# endif\r\n# if !defined(AO_HAVE_short_fetch_and_add_write)\r\n# define AO_short_fetch_and_add_write(addr, val) \\\r\n AO_short_fetch_and_add_full(addr, val)\r\n# define AO_HAVE_short_fetch_and_add_write\r\n# endif\r\n# if !defined(AO_HAVE_short_fetch_and_add_read)\r\n# define AO_short_fetch_and_add_read(addr, val) \\\r\n AO_short_fetch_and_add_full(addr, val)\r\n# define AO_HAVE_short_fetch_and_add_read\r\n# endif\r\n#endif /* AO_HAVE_short_fetch_and_add_full */\r\n\r\n#if !defined(AO_HAVE_short_fetch_and_add) && \\\r\n defined(AO_HAVE_short_fetch_and_add_release)\r\n# define AO_short_fetch_and_add(addr, val) \\\r\n AO_short_fetch_and_add_release(addr, val)\r\n# define AO_HAVE_short_fetch_and_add\r\n#endif\r\n#if !defined(AO_HAVE_short_fetch_and_add) && \\\r\n defined(AO_HAVE_short_fetch_and_add_acquire)\r\n# define AO_short_fetch_and_add(addr, val) \\\r\n AO_short_fetch_and_add_acquire(addr, val)\r\n# define AO_HAVE_short_fetch_and_add\r\n#endif\r\n#if !defined(AO_HAVE_short_fetch_and_add) && \\\r\n defined(AO_HAVE_short_fetch_and_add_write)\r\n# define AO_short_fetch_and_add(addr, val) \\\r\n AO_short_fetch_and_add_write(addr, val)\r\n# define AO_HAVE_short_fetch_and_add\r\n#endif\r\n#if !defined(AO_HAVE_short_fetch_and_add) && \\\r\n defined(AO_HAVE_short_fetch_and_add_read)\r\n# define AO_short_fetch_and_add(addr, val) \\\r\n AO_short_fetch_and_add_read(addr, val)\r\n# define AO_HAVE_short_fetch_and_add\r\n#endif\r\n\r\n#if defined(AO_HAVE_short_fetch_and_add_acquire) &&\\\r\n defined(AO_HAVE_nop_full) && \\\r\n !defined(AO_HAVE_short_fetch_and_add_full)\r\n# define AO_short_fetch_and_add_full(addr, val) \\\r\n (AO_nop_full(), AO_short_fetch_and_add_acquire(addr, val))\r\n#endif\r\n\r\n#if !defined(AO_HAVE_short_fetch_and_add_release_write) && \\\r\n defined(AO_HAVE_short_fetch_and_add_write)\r\n# define AO_short_fetch_and_add_release_write(addr, val) \\\r\n AO_short_fetch_and_add_write(addr, val)\r\n# define AO_HAVE_short_fetch_and_add_release_write\r\n#endif\r\n#if !defined(AO_HAVE_short_fetch_and_add_release_write) && \\\r\n defined(AO_HAVE_short_fetch_and_add_release)\r\n# define AO_short_fetch_and_add_release_write(addr, val) \\\r\n AO_short_fetch_and_add_release(addr, val)\r\n# define AO_HAVE_short_fetch_and_add_release_write\r\n#endif\r\n#if !defined(AO_HAVE_short_fetch_and_add_acquire_read) && \\\r\n defined(AO_HAVE_short_fetch_and_add_read)\r\n# define AO_short_fetch_and_add_acquire_read(addr, val) \\\r\n AO_short_fetch_and_add_read(addr, val)\r\n# define AO_HAVE_short_fetch_and_add_acquire_read\r\n#endif\r\n#if !defined(AO_HAVE_short_fetch_and_add_acquire_read) && \\\r\n defined(AO_HAVE_short_fetch_and_add_acquire)\r\n# define AO_short_fetch_and_add_acquire_read(addr, val) \\\r\n AO_short_fetch_and_add_acquire(addr, val)\r\n# define AO_HAVE_short_fetch_and_add_acquire_read\r\n#endif\r\n\r\n#ifdef AO_NO_DD_ORDERING\r\n# if defined(AO_HAVE_short_fetch_and_add_acquire_read)\r\n# define AO_short_fetch_and_add_dd_acquire_read(addr, val) \\\r\n AO_short_fetch_and_add_acquire_read(addr, val)\r\n# define AO_HAVE_short_fetch_and_add_dd_acquire_read\r\n# endif\r\n#else\r\n# if defined(AO_HAVE_short_fetch_and_add)\r\n# define AO_short_fetch_and_add_dd_acquire_read(addr, val) \\\r\n AO_short_fetch_and_add(addr, val)\r\n# define AO_HAVE_short_fetch_and_add_dd_acquire_read\r\n# endif\r\n#endif\r\n\r\n/* short_fetch_and_add1 */\r\n\r\n#if defined(AO_HAVE_short_fetch_and_add_full) &&\\\r\n !defined(AO_HAVE_short_fetch_and_add1_full)\r\n# define AO_short_fetch_and_add1_full(addr) \\\r\n AO_short_fetch_and_add_full(addr,1)\r\n# define AO_HAVE_short_fetch_and_add1_full\r\n#endif\r\n#if defined(AO_HAVE_short_fetch_and_add_release) &&\\\r\n !defined(AO_HAVE_short_fetch_and_add1_release)\r\n# define AO_short_fetch_and_add1_release(addr) \\\r\n AO_short_fetch_and_add_release(addr,1)\r\n# define AO_HAVE_short_fetch_and_add1_release\r\n#endif\r\n#if defined(AO_HAVE_short_fetch_and_add_acquire) &&\\\r\n !defined(AO_HAVE_short_fetch_and_add1_acquire)\r\n# define AO_short_fetch_and_add1_acquire(addr) \\\r\n AO_short_fetch_and_add_acquire(addr,1)\r\n# define AO_HAVE_short_fetch_and_add1_acquire\r\n#endif\r\n#if defined(AO_HAVE_short_fetch_and_add_write) &&\\\r\n !defined(AO_HAVE_short_fetch_and_add1_write)\r\n# define AO_short_fetch_and_add1_write(addr) \\\r\n AO_short_fetch_and_add_write(addr,1)\r\n# define AO_HAVE_short_fetch_and_add1_write\r\n#endif\r\n#if defined(AO_HAVE_short_fetch_and_add_read) &&\\\r\n !defined(AO_HAVE_short_fetch_and_add1_read)\r\n# define AO_short_fetch_and_add1_read(addr) \\\r\n AO_short_fetch_and_add_read(addr,1)\r\n# define AO_HAVE_short_fetch_and_add1_read\r\n#endif\r\n#if defined(AO_HAVE_short_fetch_and_add_release_write) &&\\\r\n !defined(AO_HAVE_short_fetch_and_add1_release_write)\r\n# define AO_short_fetch_and_add1_release_write(addr) \\\r\n AO_short_fetch_and_add_release_write(addr,1)\r\n# define AO_HAVE_short_fetch_and_add1_release_write\r\n#endif\r\n#if defined(AO_HAVE_short_fetch_and_add_acquire_read) &&\\\r\n !defined(AO_HAVE_short_fetch_and_add1_acquire_read)\r\n# define AO_short_fetch_and_add1_acquire_read(addr) \\\r\n AO_short_fetch_and_add_acquire_read(addr,1)\r\n# define AO_HAVE_short_fetch_and_add1_acquire_read\r\n#endif\r\n#if defined(AO_HAVE_short_fetch_and_add) &&\\\r\n !defined(AO_HAVE_short_fetch_and_add1)\r\n# define AO_short_fetch_and_add1(addr) \\\r\n AO_short_fetch_and_add(addr,1)\r\n# define AO_HAVE_short_fetch_and_add1\r\n#endif\r\n\r\n#if defined(AO_HAVE_short_fetch_and_add1_full)\r\n# if !defined(AO_HAVE_short_fetch_and_add1_release)\r\n# define AO_short_fetch_and_add1_release(addr) \\\r\n AO_short_fetch_and_add1_full(addr)\r\n# define AO_HAVE_short_fetch_and_add1_release\r\n# endif\r\n# if !defined(AO_HAVE_short_fetch_and_add1_acquire)\r\n# define AO_short_fetch_and_add1_acquire(addr) \\\r\n AO_short_fetch_and_add1_full(addr)\r\n# define AO_HAVE_short_fetch_and_add1_acquire\r\n# endif\r\n# if !defined(AO_HAVE_short_fetch_and_add1_write)\r\n# define AO_short_fetch_and_add1_write(addr) \\\r\n AO_short_fetch_and_add1_full(addr)\r\n# define AO_HAVE_short_fetch_and_add1_write\r\n# endif\r\n# if !defined(AO_HAVE_short_fetch_and_add1_read)\r\n# define AO_short_fetch_and_add1_read(addr) \\\r\n AO_short_fetch_and_add1_full(addr)\r\n# define AO_HAVE_short_fetch_and_add1_read\r\n# endif\r\n#endif /* AO_HAVE_short_fetch_and_add1_full */\r\n\r\n#if !defined(AO_HAVE_short_fetch_and_add1) && \\\r\n defined(AO_HAVE_short_fetch_and_add1_release)\r\n# define AO_short_fetch_and_add1(addr) \\\r\n AO_short_fetch_and_add1_release(addr)\r\n# define AO_HAVE_short_fetch_and_add1\r\n#endif\r\n#if !defined(AO_HAVE_short_fetch_and_add1) && \\\r\n defined(AO_HAVE_short_fetch_and_add1_acquire)\r\n# define AO_short_fetch_and_add1(addr) \\\r\n AO_short_fetch_and_add1_acquire(addr)\r\n# define AO_HAVE_short_fetch_and_add1\r\n#endif\r\n#if !defined(AO_HAVE_short_fetch_and_add1) && \\\r\n defined(AO_HAVE_short_fetch_and_add1_write)\r\n# define AO_short_fetch_and_add1(addr) \\\r\n AO_short_fetch_and_add1_write(addr)\r\n# define AO_HAVE_short_fetch_and_add1\r\n#endif\r\n#if !defined(AO_HAVE_short_fetch_and_add1) && \\\r\n defined(AO_HAVE_short_fetch_and_add1_read)\r\n# define AO_short_fetch_and_add1(addr) \\\r\n AO_short_fetch_and_add1_read(addr)\r\n# define AO_HAVE_short_fetch_and_add1\r\n#endif\r\n\r\n#if defined(AO_HAVE_short_fetch_and_add1_acquire) &&\\\r\n defined(AO_HAVE_nop_full) && \\\r\n !defined(AO_HAVE_short_fetch_and_add1_full)\r\n# define AO_short_fetch_and_add1_full(addr) \\\r\n (AO_nop_full(), AO_short_fetch_and_add1_acquire(addr))\r\n# define AO_HAVE_short_fetch_and_add1_full\r\n#endif\r\n\r\n#if !defined(AO_HAVE_short_fetch_and_add1_release_write) && \\\r\n defined(AO_HAVE_short_fetch_and_add1_write)\r\n# define AO_short_fetch_and_add1_release_write(addr) \\\r\n AO_short_fetch_and_add1_write(addr)\r\n# define AO_HAVE_short_fetch_and_add1_release_write\r\n#endif\r\n#if !defined(AO_HAVE_short_fetch_and_add1_release_write) && \\\r\n defined(AO_HAVE_short_fetch_and_add1_release)\r\n# define AO_short_fetch_and_add1_release_write(addr) \\\r\n AO_short_fetch_and_add1_release(addr)\r\n# define AO_HAVE_short_fetch_and_add1_release_write\r\n#endif\r\n#if !defined(AO_HAVE_short_fetch_and_add1_acquire_read) && \\\r\n defined(AO_HAVE_short_fetch_and_add1_read)\r\n# define AO_short_fetch_and_add1_acquire_read(addr) \\\r\n AO_short_fetch_and_add1_read(addr)\r\n# define AO_HAVE_short_fetch_and_add1_acquire_read\r\n#endif\r\n#if !defined(AO_HAVE_short_fetch_and_add1_acquire_read) && \\\r\n defined(AO_HAVE_short_fetch_and_add1_acquire)\r\n# define AO_short_fetch_and_add1_acquire_read(addr) \\\r\n AO_short_fetch_and_add1_acquire(addr)\r\n# define AO_HAVE_short_fetch_and_add1_acquire_read\r\n#endif\r\n\r\n#ifdef AO_NO_DD_ORDERING\r\n# if defined(AO_HAVE_short_fetch_and_add1_acquire_read)\r\n# define AO_short_fetch_and_add1_dd_acquire_read(addr) \\\r\n AO_short_fetch_and_add1_acquire_read(addr)\r\n# define AO_HAVE_short_fetch_and_add1_dd_acquire_read\r\n# endif\r\n#else\r\n# if defined(AO_HAVE_short_fetch_and_add1)\r\n# define AO_short_fetch_and_add1_dd_acquire_read(addr) \\\r\n AO_short_fetch_and_add1(addr)\r\n# define AO_HAVE_short_fetch_and_add1_dd_acquire_read\r\n# endif\r\n#endif\r\n\r\n/* short_fetch_and_sub1 */\r\n\r\n#if defined(AO_HAVE_short_fetch_and_add_full) &&\\\r\n !defined(AO_HAVE_short_fetch_and_sub1_full)\r\n# define AO_short_fetch_and_sub1_full(addr) \\\r\n AO_short_fetch_and_add_full(addr,(unsigned short)(-1))\r\n# define AO_HAVE_short_fetch_and_sub1_full\r\n#endif\r\n#if defined(AO_HAVE_short_fetch_and_add_release) &&\\\r\n !defined(AO_HAVE_short_fetch_and_sub1_release)\r\n# define AO_short_fetch_and_sub1_release(addr) \\\r\n AO_short_fetch_and_add_release(addr,(unsigned short)(-1))\r\n# define AO_HAVE_short_fetch_and_sub1_release\r\n#endif\r\n#if defined(AO_HAVE_short_fetch_and_add_acquire) &&\\\r\n !defined(AO_HAVE_short_fetch_and_sub1_acquire)\r\n# define AO_short_fetch_and_sub1_acquire(addr) \\\r\n AO_short_fetch_and_add_acquire(addr,(unsigned short)(-1))\r\n# define AO_HAVE_short_fetch_and_sub1_acquire\r\n#endif\r\n#if defined(AO_HAVE_short_fetch_and_add_write) &&\\\r\n !defined(AO_HAVE_short_fetch_and_sub1_write)\r\n# define AO_short_fetch_and_sub1_write(addr) \\\r\n AO_short_fetch_and_add_write(addr,(unsigned short)(-1))\r\n# define AO_HAVE_short_fetch_and_sub1_write\r\n#endif\r\n#if defined(AO_HAVE_short_fetch_and_add_read) &&\\\r\n !defined(AO_HAVE_short_fetch_and_sub1_read)\r\n# define AO_short_fetch_and_sub1_read(addr) \\\r\n AO_short_fetch_and_add_read(addr,(unsigned short)(-1))\r\n# define AO_HAVE_short_fetch_and_sub1_read\r\n#endif\r\n#if defined(AO_HAVE_short_fetch_and_add_release_write) &&\\\r\n !defined(AO_HAVE_short_fetch_and_sub1_release_write)\r\n# define AO_short_fetch_and_sub1_release_write(addr) \\\r\n AO_short_fetch_and_add_release_write(addr,(unsigned short)(-1))\r\n# define AO_HAVE_short_fetch_and_sub1_release_write\r\n#endif\r\n#if defined(AO_HAVE_short_fetch_and_add_acquire_read) &&\\\r\n !defined(AO_HAVE_short_fetch_and_sub1_acquire_read)\r\n# define AO_short_fetch_and_sub1_acquire_read(addr) \\\r\n AO_short_fetch_and_add_acquire_read(addr,(unsigned short)(-1))\r\n# define AO_HAVE_short_fetch_and_sub1_acquire_read\r\n#endif\r\n#if defined(AO_HAVE_short_fetch_and_add) &&\\\r\n !defined(AO_HAVE_short_fetch_and_sub1)\r\n# define AO_short_fetch_and_sub1(addr) \\\r\n AO_short_fetch_and_add(addr,(unsigned short)(-1))\r\n# define AO_HAVE_short_fetch_and_sub1\r\n#endif\r\n\r\n#if defined(AO_HAVE_short_fetch_and_sub1_full)\r\n# if !defined(AO_HAVE_short_fetch_and_sub1_release)\r\n# define AO_short_fetch_and_sub1_release(addr) \\\r\n AO_short_fetch_and_sub1_full(addr)\r\n# define AO_HAVE_short_fetch_and_sub1_release\r\n# endif\r\n# if !defined(AO_HAVE_short_fetch_and_sub1_acquire)\r\n# define AO_short_fetch_and_sub1_acquire(addr) \\\r\n AO_short_fetch_and_sub1_full(addr)\r\n# define AO_HAVE_short_fetch_and_sub1_acquire\r\n# endif\r\n# if !defined(AO_HAVE_short_fetch_and_sub1_write)\r\n# define AO_short_fetch_and_sub1_write(addr) \\\r\n AO_short_fetch_and_sub1_full(addr)\r\n# define AO_HAVE_short_fetch_and_sub1_write\r\n# endif\r\n# if !defined(AO_HAVE_short_fetch_and_sub1_read)\r\n# define AO_short_fetch_and_sub1_read(addr) \\\r\n AO_short_fetch_and_sub1_full(addr)\r\n# define AO_HAVE_short_fetch_and_sub1_read\r\n# endif\r\n#endif /* AO_HAVE_short_fetch_and_sub1_full */\r\n\r\n#if !defined(AO_HAVE_short_fetch_and_sub1) && \\\r\n defined(AO_HAVE_short_fetch_and_sub1_release)\r\n# define AO_short_fetch_and_sub1(addr) \\\r\n AO_short_fetch_and_sub1_release(addr)\r\n# define AO_HAVE_short_fetch_and_sub1\r\n#endif\r\n#if !defined(AO_HAVE_short_fetch_and_sub1) && \\\r\n defined(AO_HAVE_short_fetch_and_sub1_acquire)\r\n# define AO_short_fetch_and_sub1(addr) \\\r\n AO_short_fetch_and_sub1_acquire(addr)\r\n# define AO_HAVE_short_fetch_and_sub1\r\n#endif\r\n#if !defined(AO_HAVE_short_fetch_and_sub1) && \\\r\n defined(AO_HAVE_short_fetch_and_sub1_write)\r\n# define AO_short_fetch_and_sub1(addr) \\\r\n AO_short_fetch_and_sub1_write(addr)\r\n# define AO_HAVE_short_fetch_and_sub1\r\n#endif\r\n#if !defined(AO_HAVE_short_fetch_and_sub1) && \\\r\n defined(AO_HAVE_short_fetch_and_sub1_read)\r\n# define AO_short_fetch_and_sub1(addr) \\\r\n AO_short_fetch_and_sub1_read(addr)\r\n# define AO_HAVE_short_fetch_and_sub1\r\n#endif\r\n\r\n#if defined(AO_HAVE_short_fetch_and_sub1_acquire) &&\\\r\n defined(AO_HAVE_nop_full) && \\\r\n !defined(AO_HAVE_short_fetch_and_sub1_full)\r\n# define AO_short_fetch_and_sub1_full(addr) \\\r\n (AO_nop_full(), AO_short_fetch_and_sub1_acquire(addr))\r\n# define AO_HAVE_short_fetch_and_sub1_full\r\n#endif\r\n\r\n#if !defined(AO_HAVE_short_fetch_and_sub1_release_write) && \\\r\n defined(AO_HAVE_short_fetch_and_sub1_write)\r\n# define AO_short_fetch_and_sub1_release_write(addr) \\\r\n AO_short_fetch_and_sub1_write(addr)\r\n# define AO_HAVE_short_fetch_and_sub1_release_write\r\n#endif\r\n#if !defined(AO_HAVE_short_fetch_and_sub1_release_write) && \\\r\n defined(AO_HAVE_short_fetch_and_sub1_release)\r\n# define AO_short_fetch_and_sub1_release_write(addr) \\\r\n AO_short_fetch_and_sub1_release(addr)\r\n# define AO_HAVE_short_fetch_and_sub1_release_write\r\n#endif\r\n#if !defined(AO_HAVE_short_fetch_and_sub1_acquire_read) && \\\r\n defined(AO_HAVE_short_fetch_and_sub1_read)\r\n# define AO_short_fetch_and_sub1_acquire_read(addr) \\\r\n AO_short_fetch_and_sub1_read(addr)\r\n# define AO_HAVE_short_fetch_and_sub1_acquire_read\r\n#endif\r\n#if !defined(AO_HAVE_short_fetch_and_sub1_acquire_read) && \\\r\n defined(AO_HAVE_short_fetch_and_sub1_acquire)\r\n# define AO_short_fetch_and_sub1_acquire_read(addr) \\\r\n AO_short_fetch_and_sub1_acquire(addr)\r\n# define AO_HAVE_short_fetch_and_sub1_acquire_read\r\n#endif\r\n\r\n#ifdef AO_NO_DD_ORDERING\r\n# if defined(AO_HAVE_short_fetch_and_sub1_acquire_read)\r\n# define AO_short_fetch_and_sub1_dd_acquire_read(addr) \\\r\n AO_short_fetch_and_sub1_acquire_read(addr)\r\n# define AO_HAVE_short_fetch_and_sub1_dd_acquire_read\r\n# endif\r\n#else\r\n# if defined(AO_HAVE_short_fetch_and_sub1)\r\n# define AO_short_fetch_and_sub1_dd_acquire_read(addr) \\\r\n AO_short_fetch_and_sub1(addr)\r\n# define AO_HAVE_short_fetch_and_sub1_dd_acquire_read\r\n# endif\r\n#endif\r\n\r\n/* int_load */\r\n#if defined(AO_HAVE_int_load_acquire) && !defined(AO_HAVE_int_load)\r\n# define AO_int_load(addr) AO_int_load_acquire(addr)\r\n# define AO_HAVE_int_load\r\n#endif\r\n\r\n#if defined(AO_HAVE_int_load_full) && !defined(AO_HAVE_int_load_acquire)\r\n# define AO_int_load_acquire(addr) AO_int_load_full(addr)\r\n# define AO_HAVE_int_load_acquire\r\n#endif\r\n\r\n#if defined(AO_HAVE_int_load_full) && !defined(AO_HAVE_int_load_read)\r\n# define AO_int_load_read(addr) AO_int_load_full(addr)\r\n# define AO_HAVE_int_load_read\r\n#endif\r\n\r\n#if !defined(AO_HAVE_int_load_acquire_read) && defined(AO_HAVE_int_load_acquire)\r\n# define AO_int_load_acquire_read(addr) AO_int_load_acquire(addr)\r\n# define AO_HAVE_int_load_acquire_read\r\n#endif\r\n\r\n#if defined(AO_HAVE_int_load) && defined(AO_HAVE_nop_full) && \\\r\n !defined(AO_HAVE_int_load_acquire)\r\n AO_INLINE unsigned int\r\n AO_int_load_acquire(const volatile unsigned int *addr)\r\n {\r\n unsigned int result = AO_int_load(addr);\r\n /* Acquire barrier would be useless, since the load could be delayed */\r\n /* beyond it. */\r\n AO_nop_full();\r\n return result;\r\n }\r\n# define AO_HAVE_int_load_acquire\r\n#endif\r\n\r\n#if defined(AO_HAVE_int_load) && defined(AO_HAVE_nop_read) && \\\r\n !defined(AO_HAVE_int_load_read)\r\n AO_INLINE unsigned int\r\n AO_int_load_read(const volatile unsigned int *addr)\r\n {\r\n unsigned int result = AO_int_load(addr);\r\n /* Acquire barrier would be useless, since the load could be delayed */\r\n /* beyond it. */\r\n AO_nop_read();\r\n return result;\r\n }\r\n# define AO_HAVE_int_load_read\r\n#endif\r\n\r\n#if defined(AO_HAVE_int_load_acquire) && defined(AO_HAVE_nop_full) && \\\r\n !defined(AO_HAVE_int_load_full)\r\n# define AO_int_load_full(addr) (AO_nop_full(), AO_int_load_acquire(addr))\r\n# define AO_HAVE_int_load_full\r\n#endif\r\n\r\n#if !defined(AO_HAVE_int_load_acquire_read) && defined(AO_HAVE_int_load_read)\r\n# define AO_int_load_acquire_read(addr) AO_int_load_read(addr)\r\n# define AO_HAVE_int_load_acquire_read\r\n#endif\r\n\r\n#if defined(AO_HAVE_int_load_acquire_read) && !defined(AO_HAVE_int_load)\r\n# define AO_int_load(addr) AO_int_load_acquire_read(addr)\r\n# define AO_HAVE_int_load\r\n#endif\r\n\r\n#ifdef AO_NO_DD_ORDERING\r\n# if defined(AO_HAVE_int_load_acquire_read)\r\n# define AO_int_load_dd_acquire_read(addr) \\\r\n AO_int_load_acquire_read(addr)\r\n# define AO_HAVE_int_load_dd_acquire_read\r\n# endif\r\n#else\r\n# if defined(AO_HAVE_int_load)\r\n# define AO_int_load_dd_acquire_read(addr) \\\r\n AO_int_load(addr)\r\n# define AO_HAVE_int_load_dd_acquire_read\r\n# endif\r\n#endif\r\n\r\n\r\n/* int_store */\r\n\r\n#if defined(AO_HAVE_int_store_release) && !defined(AO_HAVE_int_store)\r\n# define AO_int_store(addr, val) AO_int_store_release(addr,val)\r\n# define AO_HAVE_int_store\r\n#endif\r\n\r\n#if defined(AO_HAVE_int_store_full) && !defined(AO_HAVE_int_store_release)\r\n# define AO_int_store_release(addr,val) AO_int_store_full(addr,val)\r\n# define AO_HAVE_int_store_release\r\n#endif\r\n\r\n#if defined(AO_HAVE_int_store_full) && !defined(AO_HAVE_int_store_write)\r\n# define AO_int_store_write(addr,val) AO_int_store_full(addr,val)\r\n# define AO_HAVE_int_store_write\r\n#endif\r\n\r\n#if defined(AO_HAVE_int_store_release) && \\\r\n !defined(AO_HAVE_int_store_release_write)\r\n# define AO_int_store_release_write(addr, val) \\\r\n AO_int_store_release(addr,val)\r\n# define AO_HAVE_int_store_release_write\r\n#endif\r\n\r\n#if defined(AO_HAVE_int_store_write) && !defined(AO_HAVE_int_store)\r\n# define AO_int_store(addr, val) AO_int_store_write(addr,val)\r\n# define AO_HAVE_int_store\r\n#endif\r\n\r\n#if defined(AO_HAVE_int_store) && defined(AO_HAVE_nop_full) && \\\r\n !defined(AO_HAVE_int_store_release)\r\n# define AO_int_store_release(addr,val) \\\r\n (AO_nop_full(), AO_int_store(addr,val))\r\n# define AO_HAVE_int_store_release\r\n#endif\r\n\r\n#if defined(AO_HAVE_nop_write) && defined(AO_HAVE_int_store) && \\\r\n !defined(AO_HAVE_int_store_write)\r\n# define AO_int_store_write(addr, val) \\\r\n (AO_nop_write(), AO_int_store(addr,val))\r\n# define AO_HAVE_int_store_write\r\n#endif\r\n\r\n#if defined(AO_HAVE_int_store_write) && \\\r\n !defined(AO_HAVE_int_store_release_write)\r\n# define AO_int_store_release_write(addr, val) AO_int_store_write(addr,val)\r\n# define AO_HAVE_int_store_release_write\r\n#endif\r\n\r\n#if defined(AO_HAVE_int_store_release) && defined(AO_HAVE_nop_full) && \\\r\n !defined(AO_HAVE_int_store_full)\r\n# define AO_int_store_full(addr, val) \\\r\n (AO_int_store_release(addr, val), AO_nop_full())\r\n# define AO_HAVE_int_store_full\r\n#endif\r\n\r\n\r\n/* int_fetch_and_add */\r\n#if defined(AO_HAVE_int_compare_and_swap_full) && \\\r\n !defined(AO_HAVE_int_fetch_and_add_full)\r\n AO_INLINE AO_t\r\n AO_int_fetch_and_add_full(volatile unsigned int *addr,\r\n unsigned int incr)\r\n {\r\n unsigned int old;\r\n do\r\n {\r\n old = *addr;\r\n }\r\n while (!AO_int_compare_and_swap_full(addr, old, old+incr));\r\n return old;\r\n }\r\n# define AO_HAVE_int_fetch_and_add_full\r\n#endif\r\n\r\n#if defined(AO_HAVE_int_compare_and_swap_acquire) && \\\r\n !defined(AO_HAVE_int_fetch_and_add_acquire)\r\n AO_INLINE AO_t\r\n AO_int_fetch_and_add_acquire(volatile unsigned int *addr,\r\n unsigned int incr)\r\n {\r\n unsigned int old;\r\n do\r\n {\r\n old = *addr;\r\n }\r\n while (!AO_int_compare_and_swap_acquire(addr, old, old+incr));\r\n return old;\r\n }\r\n# define AO_HAVE_int_fetch_and_add_acquire\r\n#endif\r\n\r\n#if defined(AO_HAVE_int_compare_and_swap_release) && \\\r\n !defined(AO_HAVE_int_fetch_and_add_release)\r\n AO_INLINE AO_t\r\n AO_int_fetch_and_add_release(volatile unsigned int *addr,\r\n unsigned int incr)\r\n {\r\n unsigned int old;\r\n do\r\n {\r\n old = *addr;\r\n }\r\n while (!AO_int_compare_and_swap_release(addr, old, old+incr));\r\n return old;\r\n }\r\n# define AO_HAVE_int_fetch_and_add_release\r\n#endif\r\n\r\n#if defined(AO_HAVE_int_fetch_and_add_full)\r\n# if !defined(AO_HAVE_int_fetch_and_add_release)\r\n# define AO_int_fetch_and_add_release(addr, val) \\\r\n AO_int_fetch_and_add_full(addr, val)\r\n# define AO_HAVE_int_fetch_and_add_release\r\n# endif\r\n# if !defined(AO_HAVE_int_fetch_and_add_acquire)\r\n# define AO_int_fetch_and_add_acquire(addr, val) \\\r\n AO_int_fetch_and_add_full(addr, val)\r\n# define AO_HAVE_int_fetch_and_add_acquire\r\n# endif\r\n# if !defined(AO_HAVE_int_fetch_and_add_write)\r\n# define AO_int_fetch_and_add_write(addr, val) \\\r\n AO_int_fetch_and_add_full(addr, val)\r\n# define AO_HAVE_int_fetch_and_add_write\r\n# endif\r\n# if !defined(AO_HAVE_int_fetch_and_add_read)\r\n# define AO_int_fetch_and_add_read(addr, val) \\\r\n AO_int_fetch_and_add_full(addr, val)\r\n# define AO_HAVE_int_fetch_and_add_read\r\n# endif\r\n#endif /* AO_HAVE_int_fetch_and_add_full */\r\n\r\n#if !defined(AO_HAVE_int_fetch_and_add) && \\\r\n defined(AO_HAVE_int_fetch_and_add_release)\r\n# define AO_int_fetch_and_add(addr, val) \\\r\n AO_int_fetch_and_add_release(addr, val)\r\n# define AO_HAVE_int_fetch_and_add\r\n#endif\r\n#if !defined(AO_HAVE_int_fetch_and_add) && \\\r\n defined(AO_HAVE_int_fetch_and_add_acquire)\r\n# define AO_int_fetch_and_add(addr, val) \\\r\n AO_int_fetch_and_add_acquire(addr, val)\r\n# define AO_HAVE_int_fetch_and_add\r\n#endif\r\n#if !defined(AO_HAVE_int_fetch_and_add) && \\\r\n defined(AO_HAVE_int_fetch_and_add_write)\r\n# define AO_int_fetch_and_add(addr, val) \\\r\n AO_int_fetch_and_add_write(addr, val)\r\n# define AO_HAVE_int_fetch_and_add\r\n#endif\r\n#if !defined(AO_HAVE_int_fetch_and_add) && \\\r\n defined(AO_HAVE_int_fetch_and_add_read)\r\n# define AO_int_fetch_and_add(addr, val) \\\r\n AO_int_fetch_and_add_read(addr, val)\r\n# define AO_HAVE_int_fetch_and_add\r\n#endif\r\n\r\n#if defined(AO_HAVE_int_fetch_and_add_acquire) &&\\\r\n defined(AO_HAVE_nop_full) && \\\r\n !defined(AO_HAVE_int_fetch_and_add_full)\r\n# define AO_int_fetch_and_add_full(addr, val) \\\r\n (AO_nop_full(), AO_int_fetch_and_add_acquire(addr, val))\r\n#endif\r\n\r\n#if !defined(AO_HAVE_int_fetch_and_add_release_write) && \\\r\n defined(AO_HAVE_int_fetch_and_add_write)\r\n# define AO_int_fetch_and_add_release_write(addr, val) \\\r\n AO_int_fetch_and_add_write(addr, val)\r\n# define AO_HAVE_int_fetch_and_add_release_write\r\n#endif\r\n#if !defined(AO_HAVE_int_fetch_and_add_release_write) && \\\r\n defined(AO_HAVE_int_fetch_and_add_release)\r\n# define AO_int_fetch_and_add_release_write(addr, val) \\\r\n AO_int_fetch_and_add_release(addr, val)\r\n# define AO_HAVE_int_fetch_and_add_release_write\r\n#endif\r\n#if !defined(AO_HAVE_int_fetch_and_add_acquire_read) && \\\r\n defined(AO_HAVE_int_fetch_and_add_read)\r\n# define AO_int_fetch_and_add_acquire_read(addr, val) \\\r\n AO_int_fetch_and_add_read(addr, val)\r\n# define AO_HAVE_int_fetch_and_add_acquire_read\r\n#endif\r\n#if !defined(AO_HAVE_int_fetch_and_add_acquire_read) && \\\r\n defined(AO_HAVE_int_fetch_and_add_acquire)\r\n# define AO_int_fetch_and_add_acquire_read(addr, val) \\\r\n AO_int_fetch_and_add_acquire(addr, val)\r\n# define AO_HAVE_int_fetch_and_add_acquire_read\r\n#endif\r\n\r\n#ifdef AO_NO_DD_ORDERING\r\n# if defined(AO_HAVE_int_fetch_and_add_acquire_read)\r\n# define AO_int_fetch_and_add_dd_acquire_read(addr, val) \\\r\n AO_int_fetch_and_add_acquire_read(addr, val)\r\n# define AO_HAVE_int_fetch_and_add_dd_acquire_read\r\n# endif\r\n#else\r\n# if defined(AO_HAVE_int_fetch_and_add)\r\n# define AO_int_fetch_and_add_dd_acquire_read(addr, val) \\\r\n AO_int_fetch_and_add(addr, val)\r\n# define AO_HAVE_int_fetch_and_add_dd_acquire_read\r\n# endif\r\n#endif\r\n\r\n/* int_fetch_and_add1 */\r\n\r\n#if defined(AO_HAVE_int_fetch_and_add_full) &&\\\r\n !defined(AO_HAVE_int_fetch_and_add1_full)\r\n# define AO_int_fetch_and_add1_full(addr) \\\r\n AO_int_fetch_and_add_full(addr,1)\r\n# define AO_HAVE_int_fetch_and_add1_full\r\n#endif\r\n#if defined(AO_HAVE_int_fetch_and_add_release) &&\\\r\n !defined(AO_HAVE_int_fetch_and_add1_release)\r\n# define AO_int_fetch_and_add1_release(addr) \\\r\n AO_int_fetch_and_add_release(addr,1)\r\n# define AO_HAVE_int_fetch_and_add1_release\r\n#endif\r\n#if defined(AO_HAVE_int_fetch_and_add_acquire) &&\\\r\n !defined(AO_HAVE_int_fetch_and_add1_acquire)\r\n# define AO_int_fetch_and_add1_acquire(addr) \\\r\n AO_int_fetch_and_add_acquire(addr,1)\r\n# define AO_HAVE_int_fetch_and_add1_acquire\r\n#endif\r\n#if defined(AO_HAVE_int_fetch_and_add_write) &&\\\r\n !defined(AO_HAVE_int_fetch_and_add1_write)\r\n# define AO_int_fetch_and_add1_write(addr) \\\r\n AO_int_fetch_and_add_write(addr,1)\r\n# define AO_HAVE_int_fetch_and_add1_write\r\n#endif\r\n#if defined(AO_HAVE_int_fetch_and_add_read) &&\\\r\n !defined(AO_HAVE_int_fetch_and_add1_read)\r\n# define AO_int_fetch_and_add1_read(addr) \\\r\n AO_int_fetch_and_add_read(addr,1)\r\n# define AO_HAVE_int_fetch_and_add1_read\r\n#endif\r\n#if defined(AO_HAVE_int_fetch_and_add_release_write) &&\\\r\n !defined(AO_HAVE_int_fetch_and_add1_release_write)\r\n# define AO_int_fetch_and_add1_release_write(addr) \\\r\n AO_int_fetch_and_add_release_write(addr,1)\r\n# define AO_HAVE_int_fetch_and_add1_release_write\r\n#endif\r\n#if defined(AO_HAVE_int_fetch_and_add_acquire_read) &&\\\r\n !defined(AO_HAVE_int_fetch_and_add1_acquire_read)\r\n# define AO_int_fetch_and_add1_acquire_read(addr) \\\r\n AO_int_fetch_and_add_acquire_read(addr,1)\r\n# define AO_HAVE_int_fetch_and_add1_acquire_read\r\n#endif\r\n#if defined(AO_HAVE_int_fetch_and_add) &&\\\r\n !defined(AO_HAVE_int_fetch_and_add1)\r\n# define AO_int_fetch_and_add1(addr) \\\r\n AO_int_fetch_and_add(addr,1)\r\n# define AO_HAVE_int_fetch_and_add1\r\n#endif\r\n\r\n#if defined(AO_HAVE_int_fetch_and_add1_full)\r\n# if !defined(AO_HAVE_int_fetch_and_add1_release)\r\n# define AO_int_fetch_and_add1_release(addr) \\\r\n AO_int_fetch_and_add1_full(addr)\r\n# define AO_HAVE_int_fetch_and_add1_release\r\n# endif\r\n# if !defined(AO_HAVE_int_fetch_and_add1_acquire)\r\n# define AO_int_fetch_and_add1_acquire(addr) \\\r\n AO_int_fetch_and_add1_full(addr)\r\n# define AO_HAVE_int_fetch_and_add1_acquire\r\n# endif\r\n# if !defined(AO_HAVE_int_fetch_and_add1_write)\r\n# define AO_int_fetch_and_add1_write(addr) \\\r\n AO_int_fetch_and_add1_full(addr)\r\n# define AO_HAVE_int_fetch_and_add1_write\r\n# endif\r\n# if !defined(AO_HAVE_int_fetch_and_add1_read)\r\n# define AO_int_fetch_and_add1_read(addr) \\\r\n AO_int_fetch_and_add1_full(addr)\r\n# define AO_HAVE_int_fetch_and_add1_read\r\n# endif\r\n#endif /* AO_HAVE_int_fetch_and_add1_full */\r\n\r\n#if !defined(AO_HAVE_int_fetch_and_add1) && \\\r\n defined(AO_HAVE_int_fetch_and_add1_release)\r\n# define AO_int_fetch_and_add1(addr) \\\r\n AO_int_fetch_and_add1_release(addr)\r\n# define AO_HAVE_int_fetch_and_add1\r\n#endif\r\n#if !defined(AO_HAVE_int_fetch_and_add1) && \\\r\n defined(AO_HAVE_int_fetch_and_add1_acquire)\r\n# define AO_int_fetch_and_add1(addr) \\\r\n AO_int_fetch_and_add1_acquire(addr)\r\n# define AO_HAVE_int_fetch_and_add1\r\n#endif\r\n#if !defined(AO_HAVE_int_fetch_and_add1) && \\\r\n defined(AO_HAVE_int_fetch_and_add1_write)\r\n# define AO_int_fetch_and_add1(addr) \\\r\n AO_int_fetch_and_add1_write(addr)\r\n# define AO_HAVE_int_fetch_and_add1\r\n#endif\r\n#if !defined(AO_HAVE_int_fetch_and_add1) && \\\r\n defined(AO_HAVE_int_fetch_and_add1_read)\r\n# define AO_int_fetch_and_add1(addr) \\\r\n AO_int_fetch_and_add1_read(addr)\r\n# define AO_HAVE_int_fetch_and_add1\r\n#endif\r\n\r\n#if defined(AO_HAVE_int_fetch_and_add1_acquire) &&\\\r\n defined(AO_HAVE_nop_full) && \\\r\n !defined(AO_HAVE_int_fetch_and_add1_full)\r\n# define AO_int_fetch_and_add1_full(addr) \\\r\n (AO_nop_full(), AO_int_fetch_and_add1_acquire(addr))\r\n# define AO_HAVE_int_fetch_and_add1_full\r\n#endif\r\n\r\n#if !defined(AO_HAVE_int_fetch_and_add1_release_write) && \\\r\n defined(AO_HAVE_int_fetch_and_add1_write)\r\n# define AO_int_fetch_and_add1_release_write(addr) \\\r\n AO_int_fetch_and_add1_write(addr)\r\n# define AO_HAVE_int_fetch_and_add1_release_write\r\n#endif\r\n#if !defined(AO_HAVE_int_fetch_and_add1_release_write) && \\\r\n defined(AO_HAVE_int_fetch_and_add1_release)\r\n# define AO_int_fetch_and_add1_release_write(addr) \\\r\n AO_int_fetch_and_add1_release(addr)\r\n# define AO_HAVE_int_fetch_and_add1_release_write\r\n#endif\r\n#if !defined(AO_HAVE_int_fetch_and_add1_acquire_read) && \\\r\n defined(AO_HAVE_int_fetch_and_add1_read)\r\n# define AO_int_fetch_and_add1_acquire_read(addr) \\\r\n AO_int_fetch_and_add1_read(addr)\r\n# define AO_HAVE_int_fetch_and_add1_acquire_read\r\n#endif\r\n#if !defined(AO_HAVE_int_fetch_and_add1_acquire_read) && \\\r\n defined(AO_HAVE_int_fetch_and_add1_acquire)\r\n# define AO_int_fetch_and_add1_acquire_read(addr) \\\r\n AO_int_fetch_and_add1_acquire(addr)\r\n# define AO_HAVE_int_fetch_and_add1_acquire_read\r\n#endif\r\n\r\n#ifdef AO_NO_DD_ORDERING\r\n# if defined(AO_HAVE_int_fetch_and_add1_acquire_read)\r\n# define AO_int_fetch_and_add1_dd_acquire_read(addr) \\\r\n AO_int_fetch_and_add1_acquire_read(addr)\r\n# define AO_HAVE_int_fetch_and_add1_dd_acquire_read\r\n# endif\r\n#else\r\n# if defined(AO_HAVE_int_fetch_and_add1)\r\n# define AO_int_fetch_and_add1_dd_acquire_read(addr) \\\r\n AO_int_fetch_and_add1(addr)\r\n# define AO_HAVE_int_fetch_and_add1_dd_acquire_read\r\n# endif\r\n#endif\r\n\r\n/* int_fetch_and_sub1 */\r\n\r\n#if defined(AO_HAVE_int_fetch_and_add_full) &&\\\r\n !defined(AO_HAVE_int_fetch_and_sub1_full)\r\n# define AO_int_fetch_and_sub1_full(addr) \\\r\n AO_int_fetch_and_add_full(addr,(unsigned int)(-1))\r\n# define AO_HAVE_int_fetch_and_sub1_full\r\n#endif\r\n#if defined(AO_HAVE_int_fetch_and_add_release) &&\\\r\n !defined(AO_HAVE_int_fetch_and_sub1_release)\r\n# define AO_int_fetch_and_sub1_release(addr) \\\r\n AO_int_fetch_and_add_release(addr,(unsigned int)(-1))\r\n# define AO_HAVE_int_fetch_and_sub1_release\r\n#endif\r\n#if defined(AO_HAVE_int_fetch_and_add_acquire) &&\\\r\n !defined(AO_HAVE_int_fetch_and_sub1_acquire)\r\n# define AO_int_fetch_and_sub1_acquire(addr) \\\r\n AO_int_fetch_and_add_acquire(addr,(unsigned int)(-1))\r\n# define AO_HAVE_int_fetch_and_sub1_acquire\r\n#endif\r\n#if defined(AO_HAVE_int_fetch_and_add_write) &&\\\r\n !defined(AO_HAVE_int_fetch_and_sub1_write)\r\n# define AO_int_fetch_and_sub1_write(addr) \\\r\n AO_int_fetch_and_add_write(addr,(unsigned int)(-1))\r\n# define AO_HAVE_int_fetch_and_sub1_write\r\n#endif\r\n#if defined(AO_HAVE_int_fetch_and_add_read) &&\\\r\n !defined(AO_HAVE_int_fetch_and_sub1_read)\r\n# define AO_int_fetch_and_sub1_read(addr) \\\r\n AO_int_fetch_and_add_read(addr,(unsigned int)(-1))\r\n# define AO_HAVE_int_fetch_and_sub1_read\r\n#endif\r\n#if defined(AO_HAVE_int_fetch_and_add_release_write) &&\\\r\n !defined(AO_HAVE_int_fetch_and_sub1_release_write)\r\n# define AO_int_fetch_and_sub1_release_write(addr) \\\r\n AO_int_fetch_and_add_release_write(addr,(unsigned int)(-1))\r\n# define AO_HAVE_int_fetch_and_sub1_release_write\r\n#endif\r\n#if defined(AO_HAVE_int_fetch_and_add_acquire_read) &&\\\r\n !defined(AO_HAVE_int_fetch_and_sub1_acquire_read)\r\n# define AO_int_fetch_and_sub1_acquire_read(addr) \\\r\n AO_int_fetch_and_add_acquire_read(addr,(unsigned int)(-1))\r\n# define AO_HAVE_int_fetch_and_sub1_acquire_read\r\n#endif\r\n#if defined(AO_HAVE_int_fetch_and_add) &&\\\r\n !defined(AO_HAVE_int_fetch_and_sub1)\r\n# define AO_int_fetch_and_sub1(addr) \\\r\n AO_int_fetch_and_add(addr,(unsigned int)(-1))\r\n# define AO_HAVE_int_fetch_and_sub1\r\n#endif\r\n\r\n#if defined(AO_HAVE_int_fetch_and_sub1_full)\r\n# if !defined(AO_HAVE_int_fetch_and_sub1_release)\r\n# define AO_int_fetch_and_sub1_release(addr) \\\r\n AO_int_fetch_and_sub1_full(addr)\r\n# define AO_HAVE_int_fetch_and_sub1_release\r\n# endif\r\n# if !defined(AO_HAVE_int_fetch_and_sub1_acquire)\r\n# define AO_int_fetch_and_sub1_acquire(addr) \\\r\n AO_int_fetch_and_sub1_full(addr)\r\n# define AO_HAVE_int_fetch_and_sub1_acquire\r\n# endif\r\n# if !defined(AO_HAVE_int_fetch_and_sub1_write)\r\n# define AO_int_fetch_and_sub1_write(addr) \\\r\n AO_int_fetch_and_sub1_full(addr)\r\n# define AO_HAVE_int_fetch_and_sub1_write\r\n# endif\r\n# if !defined(AO_HAVE_int_fetch_and_sub1_read)\r\n# define AO_int_fetch_and_sub1_read(addr) \\\r\n AO_int_fetch_and_sub1_full(addr)\r\n# define AO_HAVE_int_fetch_and_sub1_read\r\n# endif\r\n#endif /* AO_HAVE_int_fetch_and_sub1_full */\r\n\r\n#if !defined(AO_HAVE_int_fetch_and_sub1) && \\\r\n defined(AO_HAVE_int_fetch_and_sub1_release)\r\n# define AO_int_fetch_and_sub1(addr) \\\r\n AO_int_fetch_and_sub1_release(addr)\r\n# define AO_HAVE_int_fetch_and_sub1\r\n#endif\r\n#if !defined(AO_HAVE_int_fetch_and_sub1) && \\\r\n defined(AO_HAVE_int_fetch_and_sub1_acquire)\r\n# define AO_int_fetch_and_sub1(addr) \\\r\n AO_int_fetch_and_sub1_acquire(addr)\r\n# define AO_HAVE_int_fetch_and_sub1\r\n#endif\r\n#if !defined(AO_HAVE_int_fetch_and_sub1) && \\\r\n defined(AO_HAVE_int_fetch_and_sub1_write)\r\n# define AO_int_fetch_and_sub1(addr) \\\r\n AO_int_fetch_and_sub1_write(addr)\r\n# define AO_HAVE_int_fetch_and_sub1\r\n#endif\r\n#if !defined(AO_HAVE_int_fetch_and_sub1) && \\\r\n defined(AO_HAVE_int_fetch_and_sub1_read)\r\n# define AO_int_fetch_and_sub1(addr) \\\r\n AO_int_fetch_and_sub1_read(addr)\r\n# define AO_HAVE_int_fetch_and_sub1\r\n#endif\r\n\r\n#if defined(AO_HAVE_int_fetch_and_sub1_acquire) &&\\\r\n defined(AO_HAVE_nop_full) && \\\r\n !defined(AO_HAVE_int_fetch_and_sub1_full)\r\n# define AO_int_fetch_and_sub1_full(addr) \\\r\n (AO_nop_full(), AO_int_fetch_and_sub1_acquire(addr))\r\n# define AO_HAVE_int_fetch_and_sub1_full\r\n#endif\r\n\r\n#if !defined(AO_HAVE_int_fetch_and_sub1_release_write) && \\\r\n defined(AO_HAVE_int_fetch_and_sub1_write)\r\n# define AO_int_fetch_and_sub1_release_write(addr) \\\r\n AO_int_fetch_and_sub1_write(addr)\r\n# define AO_HAVE_int_fetch_and_sub1_release_write\r\n#endif\r\n#if !defined(AO_HAVE_int_fetch_and_sub1_release_write) && \\\r\n defined(AO_HAVE_int_fetch_and_sub1_release)\r\n# define AO_int_fetch_and_sub1_release_write(addr) \\\r\n AO_int_fetch_and_sub1_release(addr)\r\n# define AO_HAVE_int_fetch_and_sub1_release_write\r\n#endif\r\n#if !defined(AO_HAVE_int_fetch_and_sub1_acquire_read) && \\\r\n defined(AO_HAVE_int_fetch_and_sub1_read)\r\n# define AO_int_fetch_and_sub1_acquire_read(addr) \\\r\n AO_int_fetch_and_sub1_read(addr)\r\n# define AO_HAVE_int_fetch_and_sub1_acquire_read\r\n#endif\r\n#if !defined(AO_HAVE_int_fetch_and_sub1_acquire_read) && \\\r\n defined(AO_HAVE_int_fetch_and_sub1_acquire)\r\n# define AO_int_fetch_and_sub1_acquire_read(addr) \\\r\n AO_int_fetch_and_sub1_acquire(addr)\r\n# define AO_HAVE_int_fetch_and_sub1_acquire_read\r\n#endif\r\n\r\n#ifdef AO_NO_DD_ORDERING\r\n# if defined(AO_HAVE_int_fetch_and_sub1_acquire_read)\r\n# define AO_int_fetch_and_sub1_dd_acquire_read(addr) \\\r\n AO_int_fetch_and_sub1_acquire_read(addr)\r\n# define AO_HAVE_int_fetch_and_sub1_dd_acquire_read\r\n# endif\r\n#else\r\n# if defined(AO_HAVE_int_fetch_and_sub1)\r\n# define AO_int_fetch_and_sub1_dd_acquire_read(addr) \\\r\n AO_int_fetch_and_sub1(addr)\r\n# define AO_HAVE_int_fetch_and_sub1_dd_acquire_read\r\n# endif\r\n#endif\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/generalize.h", "content": "/*\r\n * Copyright (c) 2003-2004 Hewlett-Packard Development Company, L.P.\r\n *\r\n * Permission is hereby granted, free of charge, to any person obtaining a copy\r\n * of this software and associated documentation files (the \"Software\"), to deal\r\n * in the Software without restriction, including without limitation the rights\r\n * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\r\n * copies of the Software, and to permit persons to whom the Software is\r\n * furnished to do so, subject to the following conditions:\r\n *\r\n * The above copyright notice and this permission notice shall be included in\r\n * all copies or substantial portions of the Software.\r\n *\r\n * THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\r\n * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\r\n * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\r\n * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\r\n * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\r\n * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\r\n * SOFTWARE.\r\n */\r\n\r\n/*\r\n * Generalize atomic operations for atomic_ops.h.\r\n * Should not be included directly.\r\n *\r\n * We make no attempt to define useless operations, such as\r\n * AO_nop_acquire\r\n * AO_nop_release\r\n *\r\n * We have also so far neglected to define some others, which\r\n * do not appear likely to be useful, e.g. stores with acquire\r\n * or read barriers.\r\n *\r\n * This file is sometimes included twice by atomic_ops.h.\r\n * All definitions include explicit checks that we are not replacing\r\n * an earlier definition. In general, more desirable expansions\r\n * appear earlier so that we are more likely to use them.\r\n *\r\n * We only make safe generalizations, except that by default we define\r\n * the ...dd_acquire_read operations to be equivalent to those without\r\n * a barrier. On platforms for which this is unsafe, the platform-specific\r\n * file must define AO_NO_DD_ORDERING.\r\n */\r\n\r\n#ifndef ATOMIC_OPS_H\r\n# error Atomic_ops_generalize.h should not be included directly.\r\n#endif\r\n\r\n#if AO_CHAR_TS_T\r\n# define AO_TS_COMPARE_AND_SWAP_FULL(a,o,n) \\\r\n AO_char_compare_and_swap_full(a,o,n)\r\n# define AO_TS_COMPARE_AND_SWAP_ACQUIRE(a,o,n) \\\r\n AO_char_compare_and_swap_acquire(a,o,n)\r\n# define AO_TS_COMPARE_AND_SWAP_RELEASE(a,o,n) \\\r\n AO_char_compare_and_swap_release(a,o,n)\r\n# define AO_TS_COMPARE_AND_SWAP(a,o,n) \\\r\n AO_char_compare_and_swap(a,o,n)\r\n#endif\r\n\r\n#if AO_AO_TS_T\r\n# define AO_TS_COMPARE_AND_SWAP_FULL(a,o,n) \\\r\n AO_compare_and_swap_full(a,o,n)\r\n# define AO_TS_COMPARE_AND_SWAP_ACQUIRE(a,o,n) \\\r\n AO_compare_and_swap_acquire(a,o,n)\r\n# define AO_TS_COMPARE_AND_SWAP_RELEASE(a,o,n) \\\r\n AO_compare_and_swap_release(a,o,n)\r\n# define AO_TS_COMPARE_AND_SWAP(a,o,n) \\\r\n AO_compare_and_swap(a,o,n)\r\n#endif\r\n\r\n/* Generate test_and_set_full, if necessary and possible. */\r\n#if !defined(AO_HAVE_test_and_set) && \\\r\n !defined(AO_HAVE_test_and_set_release) && \\\r\n !defined(AO_HAVE_test_and_set_acquire) && \\\r\n !defined(AO_HAVE_test_and_set_read) && \\\r\n !defined(AO_HAVE_test_and_set_full)\r\n# if AO_AO_TS_T && defined(AO_HAVE_compare_and_swap_full) || \\\r\n AO_CHAR_TS_T && defined(AO_HAVE_char_compare_and_swap_full)\r\n AO_INLINE AO_TS_VAL_t\r\n AO_test_and_set_full(volatile AO_TS_t *addr)\r\n {\r\n if (AO_TS_COMPARE_AND_SWAP_FULL(addr, AO_TS_CLEAR, AO_TS_SET))\r\n return AO_TS_CLEAR;\r\n else\r\n return AO_TS_SET;\r\n }\r\n# define AO_HAVE_test_and_set_full\r\n# endif /* AO_HAVE_compare_and_swap_full */\r\n\r\n# if AO_AO_TS_T && defined(AO_HAVE_compare_and_swap_acquire) || \\\r\n AO_CHAR_TS_T && defined(AO_HAVE_char_compare_and_swap_acquire)\r\n AO_INLINE AO_TS_VAL_t\r\n AO_test_and_set_acquire(volatile AO_TS_t *addr)\r\n {\r\n if (AO_TS_COMPARE_AND_SWAP_ACQUIRE(addr, AO_TS_CLEAR, AO_TS_SET))\r\n return AO_TS_CLEAR;\r\n else\r\n return AO_TS_SET;\r\n }\r\n# define AO_HAVE_test_and_set_acquire\r\n# endif /* AO_HAVE_compare_and_swap_acquire */\r\n\r\n# if AO_AO_TS_T && defined(AO_HAVE_compare_and_swap_release) || \\\r\n AO_CHAR_TS_T && defined(AO_HAVE_char_compare_and_swap_release)\r\n AO_INLINE AO_TS_VAL_t\r\n AO_test_and_set_release(volatile AO_TS_t *addr)\r\n {\r\n if (AO_TS_COMPARE_AND_SWAP_RELEASE(addr, AO_TS_CLEAR, AO_TS_SET))\r\n return AO_TS_CLEAR;\r\n else\r\n return AO_TS_SET;\r\n }\r\n# define AO_HAVE_test_and_set_release\r\n# endif /* AO_HAVE_compare_and_swap_release */\r\n\r\n# if AO_AO_TS_T && defined(AO_HAVE_compare_and_swap) || \\\r\n AO_CHAR_TS_T && defined(AO_HAVE_char_compare_and_swap)\r\n AO_INLINE AO_TS_VAL_t\r\n AO_test_and_set(volatile AO_TS_t *addr)\r\n {\r\n if (AO_TS_COMPARE_AND_SWAP(addr, AO_TS_CLEAR, AO_TS_SET))\r\n return AO_TS_CLEAR;\r\n else\r\n return AO_TS_SET;\r\n }\r\n# define AO_HAVE_test_and_set\r\n# endif /* AO_HAVE_compare_and_swap */\r\n\r\n# if defined(AO_HAVE_test_and_set) && defined(AO_HAVE_nop_full) \\\r\n && !defined(AO_HAVE_test_and_set_acquire)\r\n AO_INLINE AO_TS_VAL_t\r\n AO_test_and_set_acquire(volatile AO_TS_t *addr)\r\n {\r\n AO_TS_VAL_t result = AO_test_and_set(addr);\r\n AO_nop_full();\r\n return result;\r\n }\r\n# define AO_HAVE_test_and_set_acquire\r\n# endif\r\n\r\n#endif /* No prior test and set */\r\n\r\n/* Nop */\r\n#if !defined(AO_HAVE_nop)\r\n AO_INLINE void AO_nop(void) {}\r\n# define AO_HAVE_nop\r\n#endif\r\n\r\n#if defined(AO_HAVE_test_and_set_full) && !defined(AO_HAVE_nop_full)\r\n AO_INLINE void\r\n AO_nop_full(void)\r\n {\r\n AO_TS_t dummy = AO_TS_INITIALIZER;\r\n AO_test_and_set_full(&dummy);\r\n }\r\n# define AO_HAVE_nop_full\r\n#endif\r\n\r\n#if defined(AO_HAVE_nop_acquire)\r\n# error AO_nop_acquire is useless: dont define.\r\n#endif\r\n#if defined(AO_HAVE_nop_release)\r\n# error AO_nop_release is useless: dont define.\r\n#endif\r\n\r\n#if defined(AO_HAVE_nop_full) && !defined(AO_HAVE_nop_read)\r\n# define AO_nop_read() AO_nop_full()\r\n# define AO_HAVE_nop_read\r\n#endif\r\n\r\n#if defined(AO_HAVE_nop_full) && !defined(AO_HAVE_nop_write)\r\n# define AO_nop_write() AO_nop_full()\r\n# define AO_HAVE_nop_write\r\n#endif\r\n\r\n/* Load */\r\n#if defined(AO_HAVE_load_full) && !defined(AO_HAVE_load_acquire)\r\n# define AO_load_acquire(addr) AO_load_full(addr)\r\n# define AO_HAVE_load_acquire\r\n#endif\r\n\r\n#if defined(AO_HAVE_load_acquire) && !defined(AO_HAVE_load)\r\n# define AO_load(addr) AO_load_acquire(addr)\r\n# define AO_HAVE_load\r\n#endif\r\n\r\n#if defined(AO_HAVE_load_full) && !defined(AO_HAVE_load_read)\r\n# define AO_load_read(addr) AO_load_full(addr)\r\n# define AO_HAVE_load_read\r\n#endif\r\n\r\n#if !defined(AO_HAVE_load_acquire_read) && defined(AO_HAVE_load_acquire)\r\n# define AO_load_acquire_read(addr) AO_load_acquire(addr)\r\n# define AO_HAVE_load_acquire_read\r\n#endif\r\n\r\n#if defined(AO_HAVE_load) && defined(AO_HAVE_nop_full) && \\\r\n !defined(AO_HAVE_load_acquire)\r\n AO_INLINE AO_t\r\n AO_load_acquire(const volatile AO_t *addr)\r\n {\r\n AO_t result = AO_load(addr);\r\n /* Acquire barrier would be useless, since the load could be delayed */\r\n /* beyond it. */\r\n AO_nop_full();\r\n return result;\r\n }\r\n# define AO_HAVE_load_acquire\r\n#endif\r\n\r\n#if defined(AO_HAVE_load) && defined(AO_HAVE_nop_read) && \\\r\n !defined(AO_HAVE_load_read)\r\n AO_INLINE AO_t\r\n AO_load_read(const volatile AO_t *addr)\r\n {\r\n AO_t result = AO_load(addr);\r\n /* Acquire barrier would be useless, since the load could be delayed */\r\n /* beyond it. */\r\n AO_nop_read();\r\n return result;\r\n }\r\n# define AO_HAVE_load_read\r\n#endif\r\n\r\n#if defined(AO_HAVE_load_acquire) && defined(AO_HAVE_nop_full) && \\\r\n !defined(AO_HAVE_load_full)\r\n# define AO_load_full(addr) (AO_nop_full(), AO_load_acquire(addr))\r\n# define AO_HAVE_load_full\r\n#endif\r\n\r\n#if !defined(AO_HAVE_load_acquire_read) && defined(AO_HAVE_load_read)\r\n# define AO_load_acquire_read(addr) AO_load_read(addr)\r\n# define AO_HAVE_load_acquire_read\r\n#endif\r\n\r\n#if defined(AO_HAVE_load_acquire_read) && !defined(AO_HAVE_load)\r\n# define AO_load(addr) AO_load_acquire_read(addr)\r\n# define AO_HAVE_load\r\n#endif\r\n\r\n#ifdef AO_NO_DD_ORDERING\r\n# if defined(AO_HAVE_load_acquire_read)\r\n# define AO_load_dd_acquire_read(addr) AO_load_acquire_read(addr)\r\n# define AO_HAVE_load_dd_acquire_read\r\n# endif\r\n#else\r\n# if defined(AO_HAVE_load)\r\n# define AO_load_dd_acquire_read(addr) AO_load(addr)\r\n# define AO_HAVE_load_dd_acquire_read\r\n# endif\r\n#endif\r\n\r\n\r\n/* Store */\r\n\r\n#if defined(AO_HAVE_store_full) && !defined(AO_HAVE_store_release)\r\n# define AO_store_release(addr,val) AO_store_full(addr,val)\r\n# define AO_HAVE_store_release\r\n#endif\r\n\r\n#if defined(AO_HAVE_store_release) && !defined(AO_HAVE_store)\r\n# define AO_store(addr, val) AO_store_release(addr,val)\r\n# define AO_HAVE_store\r\n#endif\r\n\r\n#if defined(AO_HAVE_store_full) && !defined(AO_HAVE_store_write)\r\n# define AO_store_write(addr,val) AO_store_full(addr,val)\r\n# define AO_HAVE_store_write\r\n#endif\r\n\r\n#if defined(AO_HAVE_store_release) && !defined(AO_HAVE_store_release_write)\r\n# define AO_store_release_write(addr, val) AO_store_release(addr,val)\r\n# define AO_HAVE_store_release_write\r\n#endif\r\n\r\n#if defined(AO_HAVE_store_write) && !defined(AO_HAVE_store)\r\n# define AO_store(addr, val) AO_store_write(addr,val)\r\n# define AO_HAVE_store\r\n#endif\r\n\r\n#if defined(AO_HAVE_store) && defined(AO_HAVE_nop_full) && \\\r\n !defined(AO_HAVE_store_release)\r\n# define AO_store_release(addr,val) (AO_nop_full(), AO_store(addr,val))\r\n# define AO_HAVE_store_release\r\n#endif\r\n\r\n#if defined(AO_HAVE_nop_write) && defined(AO_HAVE_store) && \\\r\n !defined(AO_HAVE_store_write)\r\n# define AO_store_write(addr, val) (AO_nop_write(), AO_store(addr,val))\r\n# define AO_HAVE_store_write\r\n#endif\r\n\r\n#if defined(AO_HAVE_store_write) && !defined(AO_HAVE_store_release_write)\r\n# define AO_store_release_write(addr, val) AO_store_write(addr,val)\r\n# define AO_HAVE_store_release_write\r\n#endif\r\n\r\n#if defined(AO_HAVE_store_release) && defined(AO_HAVE_nop_full) && \\\r\n !defined(AO_HAVE_store_full)\r\n# define AO_store_full(addr, val) (AO_store_release(addr, val), AO_nop_full())\r\n# define AO_HAVE_store_full\r\n#endif\r\n\r\n/* NEC LE-IT: Test and set */\r\n#if defined(AO_HAVE_test_and_set) && \\\r\n defined(AO_HAVE_nop_full) && \\\r\n !defined(AO_HAVE_test_and_set_release)\r\n# define AO_test_and_set_release(addr) \\\r\n (AO_nop_full(), AO_test_and_set(addr))\r\n# define AO_HAVE_test_and_set_release\r\n#endif\r\n\r\n#if defined(AO_HAVE_test_and_set) && \\\r\n defined(AO_HAVE_nop_full) && \\\r\n !defined(AO_HAVE_test_and_set_acquire)\r\nAO_INLINE AO_TS_t\r\nAO_test_and_set_acquire(volatile AO_TS_t *addr)\r\n{\r\n AO_TS_t res = AO_test_and_set(addr);\r\n AO_nop_full();\r\n return res;\r\n}\r\n# define AO_HAVE_test_and_set_acquire\r\n#endif\r\n\r\n\r\n/* Fetch_and_add */\r\n/* We first try to implement fetch_and_add variants in terms */\r\n/* of the corresponding compare_and_swap variants to minimize */\r\n/* adding barriers. */\r\n#if defined(AO_HAVE_compare_and_swap_full) && \\\r\n !defined(AO_HAVE_fetch_and_add_full)\r\n AO_INLINE AO_t\r\n AO_fetch_and_add_full(volatile AO_t *addr, AO_t incr)\r\n {\r\n AO_t old;\r\n do\r\n {\r\n old = *addr;\r\n }\r\n while (!AO_compare_and_swap_full(addr, old, old+incr));\r\n return old;\r\n }\r\n# define AO_HAVE_fetch_and_add_full\r\n#endif\r\n\r\n#if defined(AO_HAVE_compare_and_swap_acquire) && \\\r\n !defined(AO_HAVE_fetch_and_add_acquire)\r\n AO_INLINE AO_t\r\n AO_fetch_and_add_acquire(volatile AO_t *addr, AO_t incr)\r\n {\r\n AO_t old;\r\n do\r\n {\r\n old = *addr;\r\n }\r\n while (!AO_compare_and_swap_acquire(addr, old, old+incr));\r\n return old;\r\n }\r\n# define AO_HAVE_fetch_and_add_acquire\r\n#endif\r\n\r\n#if defined(AO_HAVE_compare_and_swap_release) && \\\r\n !defined(AO_HAVE_fetch_and_add_release)\r\n AO_INLINE AO_t\r\n AO_fetch_and_add_release(volatile AO_t *addr, AO_t incr)\r\n {\r\n AO_t old;\r\n do\r\n {\r\n old = *addr;\r\n }\r\n while (!AO_compare_and_swap_release(addr, old, old+incr));\r\n return old;\r\n }\r\n# define AO_HAVE_fetch_and_add_release\r\n#endif\r\n\r\n#if defined(AO_HAVE_compare_and_swap) && \\\r\n !defined(AO_HAVE_fetch_and_add)\r\n AO_INLINE AO_t\r\n AO_fetch_and_add(volatile AO_t *addr, AO_t incr)\r\n {\r\n AO_t old;\r\n do\r\n {\r\n old = *addr;\r\n }\r\n while (!AO_compare_and_swap(addr, old, old+incr));\r\n return old;\r\n }\r\n# define AO_HAVE_fetch_and_add\r\n#endif\r\n\r\n#if defined(AO_HAVE_fetch_and_add_full)\r\n# if !defined(AO_HAVE_fetch_and_add_release)\r\n# define AO_fetch_and_add_release(addr, val) \\\r\n AO_fetch_and_add_full(addr, val)\r\n# define AO_HAVE_fetch_and_add_release\r\n# endif\r\n# if !defined(AO_HAVE_fetch_and_add_acquire)\r\n# define AO_fetch_and_add_acquire(addr, val) \\\r\n AO_fetch_and_add_full(addr, val)\r\n# define AO_HAVE_fetch_and_add_acquire\r\n# endif\r\n# if !defined(AO_HAVE_fetch_and_add_write)\r\n# define AO_fetch_and_add_write(addr, val) \\\r\n AO_fetch_and_add_full(addr, val)\r\n# define AO_HAVE_fetch_and_add_write\r\n# endif\r\n# if !defined(AO_HAVE_fetch_and_add_read)\r\n# define AO_fetch_and_add_read(addr, val) \\\r\n AO_fetch_and_add_full(addr, val)\r\n# define AO_HAVE_fetch_and_add_read\r\n# endif\r\n#endif /* AO_HAVE_fetch_and_add_full */\r\n\r\n#if !defined(AO_HAVE_fetch_and_add) && \\\r\n defined(AO_HAVE_fetch_and_add_release)\r\n# define AO_fetch_and_add(addr, val) \\\r\n AO_fetch_and_add_release(addr, val)\r\n# define AO_HAVE_fetch_and_add\r\n#endif\r\n#if !defined(AO_HAVE_fetch_and_add) && \\\r\n defined(AO_HAVE_fetch_and_add_acquire)\r\n# define AO_fetch_and_add(addr, val) \\\r\n AO_fetch_and_add_acquire(addr, val)\r\n# define AO_HAVE_fetch_and_add\r\n#endif\r\n#if !defined(AO_HAVE_fetch_and_add) && \\\r\n defined(AO_HAVE_fetch_and_add_write)\r\n# define AO_fetch_and_add(addr, val) \\\r\n AO_fetch_and_add_write(addr, val)\r\n# define AO_HAVE_fetch_and_add\r\n#endif\r\n#if !defined(AO_HAVE_fetch_and_add) && \\\r\n defined(AO_HAVE_fetch_and_add_read)\r\n# define AO_fetch_and_add(addr, val) \\\r\n AO_fetch_and_add_read(addr, val)\r\n# define AO_HAVE_fetch_and_add\r\n#endif\r\n\r\n#if defined(AO_HAVE_fetch_and_add_acquire) &&\\\r\n defined(AO_HAVE_nop_full) && \\\r\n !defined(AO_HAVE_fetch_and_add_full)\r\n# define AO_fetch_and_add_full(addr, val) \\\r\n (AO_nop_full(), AO_fetch_and_add_acquire(addr, val))\r\n# define AO_HAVE_fetch_and_add_full\r\n#endif\r\n\r\n#if !defined(AO_HAVE_fetch_and_add_release_write) && \\\r\n defined(AO_HAVE_fetch_and_add_write)\r\n# define AO_fetch_and_add_release_write(addr, val) \\\r\n AO_fetch_and_add_write(addr, val)\r\n# define AO_HAVE_fetch_and_add_release_write\r\n#endif\r\n#if !defined(AO_HAVE_fetch_and_add_release_write) && \\\r\n defined(AO_HAVE_fetch_and_add_release)\r\n# define AO_fetch_and_add_release_write(addr, val) \\\r\n AO_fetch_and_add_release(addr, val)\r\n# define AO_HAVE_fetch_and_add_release_write\r\n#endif\r\n#if !defined(AO_HAVE_fetch_and_add_acquire_read) && \\\r\n defined(AO_HAVE_fetch_and_add_read)\r\n# define AO_fetch_and_add_acquire_read(addr, val) \\\r\n AO_fetch_and_add_read(addr, val)\r\n# define AO_HAVE_fetch_and_add_acquire_read\r\n#endif\r\n#if !defined(AO_HAVE_fetch_and_add_acquire_read) && \\\r\n defined(AO_HAVE_fetch_and_add_acquire)\r\n# define AO_fetch_and_add_acquire_read(addr, val) \\\r\n AO_fetch_and_add_acquire(addr, val)\r\n# define AO_HAVE_fetch_and_add_acquire_read\r\n#endif\r\n\r\n#ifdef AO_NO_DD_ORDERING\r\n# if defined(AO_HAVE_fetch_and_add_acquire_read)\r\n# define AO_fetch_and_add_dd_acquire_read(addr, val) \\\r\n AO_fetch_and_add_acquire_read(addr, val)\r\n# define AO_HAVE_fetch_and_add_dd_acquire_read\r\n# endif\r\n#else\r\n# if defined(AO_HAVE_fetch_and_add)\r\n# define AO_fetch_and_add_dd_acquire_read(addr, val) \\\r\n AO_fetch_and_add(addr, val)\r\n# define AO_HAVE_fetch_and_add_dd_acquire_read\r\n# endif\r\n#endif\r\n\r\n/* Fetch_and_add1 */\r\n\r\n#if defined(AO_HAVE_fetch_and_add_full) &&\\\r\n !defined(AO_HAVE_fetch_and_add1_full)\r\n# define AO_fetch_and_add1_full(addr) AO_fetch_and_add_full(addr,1)\r\n# define AO_HAVE_fetch_and_add1_full\r\n#endif\r\n#if defined(AO_HAVE_fetch_and_add_release) &&\\\r\n !defined(AO_HAVE_fetch_and_add1_release)\r\n# define AO_fetch_and_add1_release(addr) AO_fetch_and_add_release(addr,1)\r\n# define AO_HAVE_fetch_and_add1_release\r\n#endif\r\n#if defined(AO_HAVE_fetch_and_add_acquire) &&\\\r\n !defined(AO_HAVE_fetch_and_add1_acquire)\r\n# define AO_fetch_and_add1_acquire(addr) AO_fetch_and_add_acquire(addr,1)\r\n# define AO_HAVE_fetch_and_add1_acquire\r\n#endif\r\n#if defined(AO_HAVE_fetch_and_add_write) &&\\\r\n !defined(AO_HAVE_fetch_and_add1_write)\r\n# define AO_fetch_and_add1_write(addr) AO_fetch_and_add_write(addr,1)\r\n# define AO_HAVE_fetch_and_add1_write\r\n#endif\r\n#if defined(AO_HAVE_fetch_and_add_read) &&\\\r\n !defined(AO_HAVE_fetch_and_add1_read)\r\n# define AO_fetch_and_add1_read(addr) AO_fetch_and_add_read(addr,1)\r\n# define AO_HAVE_fetch_and_add1_read\r\n#endif\r\n#if defined(AO_HAVE_fetch_and_add_release_write) &&\\\r\n !defined(AO_HAVE_fetch_and_add1_release_write)\r\n# define AO_fetch_and_add1_release_write(addr) \\\r\n AO_fetch_and_add_release_write(addr,1)\r\n# define AO_HAVE_fetch_and_add1_release_write\r\n#endif\r\n#if defined(AO_HAVE_fetch_and_add_acquire_read) &&\\\r\n !defined(AO_HAVE_fetch_and_add1_acquire_read)\r\n# define AO_fetch_and_add1_acquire_read(addr) \\\r\n AO_fetch_and_add_acquire_read(addr,1)\r\n# define AO_HAVE_fetch_and_add1_acquire_read\r\n#endif\r\n#if defined(AO_HAVE_fetch_and_add) &&\\\r\n !defined(AO_HAVE_fetch_and_add1)\r\n# define AO_fetch_and_add1(addr) \\\r\n AO_fetch_and_add(addr,1)\r\n# define AO_HAVE_fetch_and_add1\r\n#endif\r\n\r\n#if defined(AO_HAVE_fetch_and_add1_full)\r\n# if !defined(AO_HAVE_fetch_and_add1_release)\r\n# define AO_fetch_and_add1_release(addr) \\\r\n AO_fetch_and_add1_full(addr)\r\n# define AO_HAVE_fetch_and_add1_release\r\n# endif\r\n# if !defined(AO_HAVE_fetch_and_add1_acquire)\r\n# define AO_fetch_and_add1_acquire(addr) \\\r\n AO_fetch_and_add1_full(addr)\r\n# define AO_HAVE_fetch_and_add1_acquire\r\n# endif\r\n# if !defined(AO_HAVE_fetch_and_add1_write)\r\n# define AO_fetch_and_add1_write(addr) \\\r\n AO_fetch_and_add1_full(addr)\r\n# define AO_HAVE_fetch_and_add1_write\r\n# endif\r\n# if !defined(AO_HAVE_fetch_and_add1_read)\r\n# define AO_fetch_and_add1_read(addr) \\\r\n AO_fetch_and_add1_full(addr)\r\n# define AO_HAVE_fetch_and_add1_read\r\n# endif\r\n#endif /* AO_HAVE_fetch_and_add1_full */\r\n\r\n#if !defined(AO_HAVE_fetch_and_add1) && \\\r\n defined(AO_HAVE_fetch_and_add1_release)\r\n# define AO_fetch_and_add1(addr) \\\r\n AO_fetch_and_add1_release(addr)\r\n# define AO_HAVE_fetch_and_add1\r\n#endif\r\n#if !defined(AO_HAVE_fetch_and_add1) && \\\r\n defined(AO_HAVE_fetch_and_add1_acquire)\r\n# define AO_fetch_and_add1(addr) \\\r\n AO_fetch_and_add1_acquire(addr)\r\n# define AO_HAVE_fetch_and_add1\r\n#endif\r\n#if !defined(AO_HAVE_fetch_and_add1) && \\\r\n defined(AO_HAVE_fetch_and_add1_write)\r\n# define AO_fetch_and_add1(addr) \\\r\n AO_fetch_and_add1_write(addr)\r\n# define AO_HAVE_fetch_and_add1\r\n#endif\r\n#if !defined(AO_HAVE_fetch_and_add1) && \\\r\n defined(AO_HAVE_fetch_and_add1_read)\r\n# define AO_fetch_and_add1(addr) \\\r\n AO_fetch_and_add1_read(addr)\r\n# define AO_HAVE_fetch_and_add1\r\n#endif\r\n\r\n#if defined(AO_HAVE_fetch_and_add1_acquire) &&\\\r\n defined(AO_HAVE_nop_full) && \\\r\n !defined(AO_HAVE_fetch_and_add1_full)\r\n# define AO_fetch_and_add1_full(addr) \\\r\n (AO_nop_full(), AO_fetch_and_add1_acquire(addr))\r\n# define AO_HAVE_fetch_and_add1_full\r\n#endif\r\n\r\n#if !defined(AO_HAVE_fetch_and_add1_release_write) && \\\r\n defined(AO_HAVE_fetch_and_add1_write)\r\n# define AO_fetch_and_add1_release_write(addr) \\\r\n AO_fetch_and_add1_write(addr)\r\n# define AO_HAVE_fetch_and_add1_release_write\r\n#endif\r\n#if !defined(AO_HAVE_fetch_and_add1_release_write) && \\\r\n defined(AO_HAVE_fetch_and_add1_release)\r\n# define AO_fetch_and_add1_release_write(addr) \\\r\n AO_fetch_and_add1_release(addr)\r\n# define AO_HAVE_fetch_and_add1_release_write\r\n#endif\r\n#if !defined(AO_HAVE_fetch_and_add1_acquire_read) && \\\r\n defined(AO_HAVE_fetch_and_add1_read)\r\n# define AO_fetch_and_add1_acquire_read(addr) \\\r\n AO_fetch_and_add1_read(addr)\r\n# define AO_HAVE_fetch_and_add1_acquire_read\r\n#endif\r\n#if !defined(AO_HAVE_fetch_and_add1_acquire_read) && \\\r\n defined(AO_HAVE_fetch_and_add1_acquire)\r\n# define AO_fetch_and_add1_acquire_read(addr) \\\r\n AO_fetch_and_add1_acquire(addr)\r\n# define AO_HAVE_fetch_and_add1_acquire_read\r\n#endif\r\n\r\n#ifdef AO_NO_DD_ORDERING\r\n# if defined(AO_HAVE_fetch_and_add1_acquire_read)\r\n# define AO_fetch_and_add1_dd_acquire_read(addr) \\\r\n AO_fetch_and_add1_acquire_read(addr)\r\n# define AO_HAVE_fetch_and_add1_dd_acquire_read\r\n# endif\r\n#else\r\n# if defined(AO_HAVE_fetch_and_add1)\r\n# define AO_fetch_and_add1_dd_acquire_read(addr) AO_fetch_and_add1(addr)\r\n# define AO_HAVE_fetch_and_add1_dd_acquire_read\r\n# endif\r\n#endif\r\n\r\n/* Fetch_and_sub1 */\r\n\r\n#if defined(AO_HAVE_fetch_and_add_full) &&\\\r\n !defined(AO_HAVE_fetch_and_sub1_full)\r\n# define AO_fetch_and_sub1_full(addr) AO_fetch_and_add_full(addr,(AO_t)(-1))\r\n# define AO_HAVE_fetch_and_sub1_full\r\n#endif\r\n#if defined(AO_HAVE_fetch_and_add_release) &&\\\r\n !defined(AO_HAVE_fetch_and_sub1_release)\r\n# define AO_fetch_and_sub1_release(addr) \\\r\n AO_fetch_and_add_release(addr,(AO_t)(-1))\r\n# define AO_HAVE_fetch_and_sub1_release\r\n#endif\r\n#if defined(AO_HAVE_fetch_and_add_acquire) &&\\\r\n !defined(AO_HAVE_fetch_and_sub1_acquire)\r\n# define AO_fetch_and_sub1_acquire(addr) \\\r\n AO_fetch_and_add_acquire(addr,(AO_t)(-1))\r\n# define AO_HAVE_fetch_and_sub1_acquire\r\n#endif\r\n#if defined(AO_HAVE_fetch_and_add_write) &&\\\r\n !defined(AO_HAVE_fetch_and_sub1_write)\r\n# define AO_fetch_and_sub1_write(addr) \\\r\n AO_fetch_and_add_write(addr,(AO_t)(-1))\r\n# define AO_HAVE_fetch_and_sub1_write\r\n#endif\r\n#if defined(AO_HAVE_fetch_and_add_read) &&\\\r\n !defined(AO_HAVE_fetch_and_sub1_read)\r\n# define AO_fetch_and_sub1_read(addr) \\\r\n AO_fetch_and_add_read(addr,(AO_t)(-1))\r\n# define AO_HAVE_fetch_and_sub1_read\r\n#endif\r\n#if defined(AO_HAVE_fetch_and_add_release_write) &&\\\r\n !defined(AO_HAVE_fetch_and_sub1_release_write)\r\n# define AO_fetch_and_sub1_release_write(addr) \\\r\n AO_fetch_and_add_release_write(addr,(AO_t)(-1))\r\n# define AO_HAVE_fetch_and_sub1_release_write\r\n#endif\r\n#if defined(AO_HAVE_fetch_and_add_acquire_read) &&\\\r\n !defined(AO_HAVE_fetch_and_sub1_acquire_read)\r\n# define AO_fetch_and_sub1_acquire_read(addr) \\\r\n AO_fetch_and_add_acquire_read(addr,(AO_t)(-1))\r\n# define AO_HAVE_fetch_and_sub1_acquire_read\r\n#endif\r\n#if defined(AO_HAVE_fetch_and_add) &&\\\r\n !defined(AO_HAVE_fetch_and_sub1)\r\n# define AO_fetch_and_sub1(addr) \\\r\n AO_fetch_and_add(addr,(AO_t)(-1))\r\n# define AO_HAVE_fetch_and_sub1\r\n#endif\r\n\r\n#if defined(AO_HAVE_fetch_and_sub1_full)\r\n# if !defined(AO_HAVE_fetch_and_sub1_release)\r\n# define AO_fetch_and_sub1_release(addr) \\\r\n AO_fetch_and_sub1_full(addr)\r\n# define AO_HAVE_fetch_and_sub1_release\r\n# endif\r\n# if !defined(AO_HAVE_fetch_and_sub1_acquire)\r\n# define AO_fetch_and_sub1_acquire(addr) \\\r\n AO_fetch_and_sub1_full(addr)\r\n# define AO_HAVE_fetch_and_sub1_acquire\r\n# endif\r\n# if !defined(AO_HAVE_fetch_and_sub1_write)\r\n# define AO_fetch_and_sub1_write(addr) \\\r\n AO_fetch_and_sub1_full(addr)\r\n# define AO_HAVE_fetch_and_sub1_write\r\n# endif\r\n# if !defined(AO_HAVE_fetch_and_sub1_read)\r\n# define AO_fetch_and_sub1_read(addr) \\\r\n AO_fetch_and_sub1_full(addr)\r\n# define AO_HAVE_fetch_and_sub1_read\r\n# endif\r\n#endif /* AO_HAVE_fetch_and_sub1_full */\r\n\r\n#if !defined(AO_HAVE_fetch_and_sub1) && \\\r\n defined(AO_HAVE_fetch_and_sub1_release)\r\n# define AO_fetch_and_sub1(addr) \\\r\n AO_fetch_and_sub1_release(addr)\r\n# define AO_HAVE_fetch_and_sub1\r\n#endif\r\n#if !defined(AO_HAVE_fetch_and_sub1) && \\\r\n defined(AO_HAVE_fetch_and_sub1_acquire)\r\n# define AO_fetch_and_sub1(addr) \\\r\n AO_fetch_and_sub1_acquire(addr)\r\n# define AO_HAVE_fetch_and_sub1\r\n#endif\r\n#if !defined(AO_HAVE_fetch_and_sub1) && \\\r\n defined(AO_HAVE_fetch_and_sub1_write)\r\n# define AO_fetch_and_sub1(addr) \\\r\n AO_fetch_and_sub1_write(addr)\r\n# define AO_HAVE_fetch_and_sub1\r\n#endif\r\n#if !defined(AO_HAVE_fetch_and_sub1) && \\\r\n defined(AO_HAVE_fetch_and_sub1_read)\r\n# define AO_fetch_and_sub1(addr) \\\r\n AO_fetch_and_sub1_read(addr)\r\n# define AO_HAVE_fetch_and_sub1\r\n#endif\r\n\r\n#if defined(AO_HAVE_fetch_and_sub1_acquire) &&\\\r\n defined(AO_HAVE_nop_full) && \\\r\n !defined(AO_HAVE_fetch_and_sub1_full)\r\n# define AO_fetch_and_sub1_full(addr) \\\r\n (AO_nop_full(), AO_fetch_and_sub1_acquire(addr))\r\n# define AO_HAVE_fetch_and_sub1_full\r\n#endif\r\n\r\n#if !defined(AO_HAVE_fetch_and_sub1_release_write) && \\\r\n defined(AO_HAVE_fetch_and_sub1_write)\r\n# define AO_fetch_and_sub1_release_write(addr) \\\r\n AO_fetch_and_sub1_write(addr)\r\n# define AO_HAVE_fetch_and_sub1_release_write\r\n#endif\r\n#if !defined(AO_HAVE_fetch_and_sub1_release_write) && \\\r\n defined(AO_HAVE_fetch_and_sub1_release)\r\n# define AO_fetch_and_sub1_release_write(addr) \\\r\n AO_fetch_and_sub1_release(addr)\r\n# define AO_HAVE_fetch_and_sub1_release_write\r\n#endif\r\n#if !defined(AO_HAVE_fetch_and_sub1_acquire_read) && \\\r\n defined(AO_HAVE_fetch_and_sub1_read)\r\n# define AO_fetch_and_sub1_acquire_read(addr) \\\r\n AO_fetch_and_sub1_read(addr)\r\n# define AO_HAVE_fetch_and_sub1_acquire_read\r\n#endif\r\n#if !defined(AO_HAVE_fetch_and_sub1_acquire_read) && \\\r\n defined(AO_HAVE_fetch_and_sub1_acquire)\r\n# define AO_fetch_and_sub1_acquire_read(addr) \\\r\n AO_fetch_and_sub1_acquire(addr)\r\n# define AO_HAVE_fetch_and_sub1_acquire_read\r\n#endif\r\n\r\n#ifdef AO_NO_DD_ORDERING\r\n# if defined(AO_HAVE_fetch_and_sub1_acquire_read)\r\n# define AO_fetch_and_sub1_dd_acquire_read(addr) \\\r\n AO_fetch_and_sub1_acquire_read(addr)\r\n# define AO_HAVE_fetch_and_sub1_dd_acquire_read\r\n# endif\r\n#else\r\n# if defined(AO_HAVE_fetch_and_sub1)\r\n# define AO_fetch_and_sub1_dd_acquire_read(addr) AO_fetch_and_sub1(addr)\r\n# define AO_HAVE_fetch_and_sub1_dd_acquire_read\r\n# endif\r\n#endif\r\n\r\n/* Atomic or */\r\n#if defined(AO_HAVE_compare_and_swap_full) && \\\r\n !defined(AO_HAVE_or_full)\r\n AO_INLINE void\r\n AO_or_full(volatile AO_t *addr, AO_t incr)\r\n {\r\n AO_t old;\r\n do\r\n {\r\n old = *addr;\r\n }\r\n while (!AO_compare_and_swap_full(addr, old, (old | incr)));\r\n }\r\n# define AO_HAVE_or_full\r\n#endif\r\n\r\n#if defined(AO_HAVE_or_full)\r\n# if !defined(AO_HAVE_or_release)\r\n# define AO_or_release(addr, val) \\\r\n AO_or_full(addr, val)\r\n# define AO_HAVE_or_release\r\n# endif\r\n# if !defined(AO_HAVE_or_acquire)\r\n# define AO_or_acquire(addr, val) \\\r\n AO_or_full(addr, val)\r\n# define AO_HAVE_or_acquire\r\n# endif\r\n# if !defined(AO_HAVE_or_write)\r\n# define AO_or_write(addr, val) \\\r\n AO_or_full(addr, val)\r\n# define AO_HAVE_or_write\r\n# endif\r\n# if !defined(AO_HAVE_or_read)\r\n# define AO_or_read(addr, val) \\\r\n AO_or_full(addr, val)\r\n# define AO_HAVE_or_read\r\n# endif\r\n#endif /* AO_HAVE_or_full */\r\n\r\n#if !defined(AO_HAVE_or) && \\\r\n defined(AO_HAVE_or_release)\r\n# define AO_or(addr, val) \\\r\n AO_or_release(addr, val)\r\n# define AO_HAVE_or\r\n#endif\r\n#if !defined(AO_HAVE_or) && \\\r\n defined(AO_HAVE_or_acquire)\r\n# define AO_or(addr, val) \\\r\n AO_or_acquire(addr, val)\r\n# define AO_HAVE_or\r\n#endif\r\n#if !defined(AO_HAVE_or) && \\\r\n defined(AO_HAVE_or_write)\r\n# define AO_or(addr, val) \\\r\n AO_or_write(addr, val)\r\n# define AO_HAVE_or\r\n#endif\r\n#if !defined(AO_HAVE_or) && \\\r\n defined(AO_HAVE_or_read)\r\n# define AO_or(addr, val) \\\r\n AO_or_read(addr, val)\r\n# define AO_HAVE_or\r\n#endif\r\n\r\n#if defined(AO_HAVE_or_acquire) &&\\\r\n defined(AO_HAVE_nop_full) && \\\r\n !defined(AO_HAVE_or_full)\r\n# define AO_or_full(addr, val) \\\r\n (AO_nop_full(), AO_or_acquire(addr, val))\r\n#endif\r\n\r\n#if !defined(AO_HAVE_or_release_write) && \\\r\n defined(AO_HAVE_or_write)\r\n# define AO_or_release_write(addr, val) \\\r\n AO_or_write(addr, val)\r\n# define AO_HAVE_or_release_write\r\n#endif\r\n#if !defined(AO_HAVE_or_release_write) && \\\r\n defined(AO_HAVE_or_release)\r\n# define AO_or_release_write(addr, val) \\\r\n AO_or_release(addr, val)\r\n# define AO_HAVE_or_release_write\r\n#endif\r\n#if !defined(AO_HAVE_or_acquire_read) && \\\r\n defined(AO_HAVE_or_read)\r\n# define AO_or_acquire_read(addr, val) \\\r\n AO_or_read(addr, val)\r\n# define AO_HAVE_or_acquire_read\r\n#endif\r\n#if !defined(AO_HAVE_or_acquire_read) && \\\r\n defined(AO_HAVE_or_acquire)\r\n# define AO_or_acquire_read(addr, val) \\\r\n AO_or_acquire(addr, val)\r\n# define AO_HAVE_or_acquire_read\r\n#endif\r\n\r\n/* dd_aquire_read is meaningless. */\r\n\r\n/* Test_and_set */\r\n\r\n#if defined(AO_HAVE_test_and_set_full)\r\n# if !defined(AO_HAVE_test_and_set_release)\r\n# define AO_test_and_set_release(addr) \\\r\n AO_test_and_set_full(addr)\r\n# define AO_HAVE_test_and_set_release\r\n# endif\r\n# if !defined(AO_HAVE_test_and_set_acquire)\r\n# define AO_test_and_set_acquire(addr) \\\r\n AO_test_and_set_full(addr)\r\n# define AO_HAVE_test_and_set_acquire\r\n# endif\r\n# if !defined(AO_HAVE_test_and_set_write)\r\n# define AO_test_and_set_write(addr) \\\r\n AO_test_and_set_full(addr)\r\n# define AO_HAVE_test_and_set_write\r\n# endif\r\n# if !defined(AO_HAVE_test_and_set_read)\r\n# define AO_test_and_set_read(addr) \\\r\n AO_test_and_set_full(addr)\r\n# define AO_HAVE_test_and_set_read\r\n# endif\r\n#endif /* AO_HAVE_test_and_set_full */\r\n\r\n#if !defined(AO_HAVE_test_and_set) && \\\r\n defined(AO_HAVE_test_and_set_release)\r\n# define AO_test_and_set(addr) \\\r\n AO_test_and_set_release(addr)\r\n# define AO_HAVE_test_and_set\r\n#endif\r\n#if !defined(AO_HAVE_test_and_set) && \\\r\n defined(AO_HAVE_test_and_set_acquire)\r\n# define AO_test_and_set(addr) \\\r\n AO_test_and_set_acquire(addr)\r\n# define AO_HAVE_test_and_set\r\n#endif\r\n#if !defined(AO_HAVE_test_and_set) && \\\r\n defined(AO_HAVE_test_and_set_write)\r\n# define AO_test_and_set(addr) \\\r\n AO_test_and_set_write(addr)\r\n# define AO_HAVE_test_and_set\r\n#endif\r\n#if !defined(AO_HAVE_test_and_set) && \\\r\n defined(AO_HAVE_test_and_set_read)\r\n# define AO_test_and_set(addr) \\\r\n AO_test_and_set_read(addr)\r\n# define AO_HAVE_test_and_set\r\n#endif\r\n\r\n#if defined(AO_HAVE_test_and_set_acquire) &&\\\r\n defined(AO_HAVE_nop_full) && \\\r\n !defined(AO_HAVE_test_and_set_full)\r\n# define AO_test_and_set_full(addr) \\\r\n (AO_nop_full(), AO_test_and_set_acquire(addr))\r\n# define AO_HAVE_test_and_set_full\r\n#endif\r\n\r\n#if !defined(AO_HAVE_test_and_set_release_write) && \\\r\n defined(AO_HAVE_test_and_set_write)\r\n# define AO_test_and_set_release_write(addr) \\\r\n AO_test_and_set_write(addr)\r\n# define AO_HAVE_test_and_set_release_write\r\n#endif\r\n#if !defined(AO_HAVE_test_and_set_release_write) && \\\r\n defined(AO_HAVE_test_and_set_release)\r\n# define AO_test_and_set_release_write(addr) \\\r\n AO_test_and_set_release(addr)\r\n# define AO_HAVE_test_and_set_release_write\r\n#endif\r\n#if !defined(AO_HAVE_test_and_set_acquire_read) && \\\r\n defined(AO_HAVE_test_and_set_read)\r\n# define AO_test_and_set_acquire_read(addr) \\\r\n AO_test_and_set_read(addr)\r\n# define AO_HAVE_test_and_set_acquire_read\r\n#endif\r\n#if !defined(AO_HAVE_test_and_set_acquire_read) && \\\r\n defined(AO_HAVE_test_and_set_acquire)\r\n# define AO_test_and_set_acquire_read(addr) \\\r\n AO_test_and_set_acquire(addr)\r\n# define AO_HAVE_test_and_set_acquire_read\r\n#endif\r\n\r\n#ifdef AO_NO_DD_ORDERING\r\n# if defined(AO_HAVE_test_and_set_acquire_read)\r\n# define AO_test_and_set_dd_acquire_read(addr) \\\r\n AO_test_and_set_acquire_read(addr)\r\n# define AO_HAVE_test_and_set_dd_acquire_read\r\n# endif\r\n#else\r\n# if defined(AO_HAVE_test_and_set)\r\n# define AO_test_and_set_dd_acquire_read(addr) AO_test_and_set(addr)\r\n# define AO_HAVE_test_and_set_dd_acquire_read\r\n# endif\r\n#endif\r\n\r\n/* Compare_and_swap */\r\n#if defined(AO_HAVE_compare_and_swap) && defined(AO_HAVE_nop_full)\\\r\n && !defined(AO_HAVE_compare_and_swap_acquire)\r\n AO_INLINE int\r\n AO_compare_and_swap_acquire(volatile AO_t *addr, AO_t old, AO_t new_val)\r\n {\r\n int result = AO_compare_and_swap(addr, old, new_val);\r\n AO_nop_full();\r\n return result;\r\n }\r\n# define AO_HAVE_compare_and_swap_acquire\r\n#endif\r\n#if defined(AO_HAVE_compare_and_swap) && defined(AO_HAVE_nop_full)\\\r\n && !defined(AO_HAVE_compare_and_swap_release)\r\n# define AO_compare_and_swap_release(addr, old, new_val) \\\r\n (AO_nop_full(), AO_compare_and_swap(addr, old, new_val))\r\n# define AO_HAVE_compare_and_swap_release\r\n#endif\r\n#if defined(AO_HAVE_compare_and_swap_full)\r\n# if !defined(AO_HAVE_compare_and_swap_release)\r\n# define AO_compare_and_swap_release(addr, old, new_val) \\\r\n AO_compare_and_swap_full(addr, old, new_val)\r\n# define AO_HAVE_compare_and_swap_release\r\n# endif\r\n# if !defined(AO_HAVE_compare_and_swap_acquire)\r\n# define AO_compare_and_swap_acquire(addr, old, new_val) \\\r\n AO_compare_and_swap_full(addr, old, new_val)\r\n# define AO_HAVE_compare_and_swap_acquire\r\n# endif\r\n# if !defined(AO_HAVE_compare_and_swap_write)\r\n# define AO_compare_and_swap_write(addr, old, new_val) \\\r\n AO_compare_and_swap_full(addr, old, new_val)\r\n# define AO_HAVE_compare_and_swap_write\r\n# endif\r\n# if !defined(AO_HAVE_compare_and_swap_read)\r\n# define AO_compare_and_swap_read(addr, old, new_val) \\\r\n AO_compare_and_swap_full(addr, old, new_val)\r\n# define AO_HAVE_compare_and_swap_read\r\n# endif\r\n#endif /* AO_HAVE_compare_and_swap_full */\r\n\r\n#if !defined(AO_HAVE_compare_and_swap) && \\\r\n defined(AO_HAVE_compare_and_swap_release)\r\n# define AO_compare_and_swap(addr, old, new_val) \\\r\n AO_compare_and_swap_release(addr, old, new_val)\r\n# define AO_HAVE_compare_and_swap\r\n#endif\r\n#if !defined(AO_HAVE_compare_and_swap) && \\\r\n defined(AO_HAVE_compare_and_swap_acquire)\r\n# define AO_compare_and_swap(addr, old, new_val) \\\r\n AO_compare_and_swap_acquire(addr, old, new_val)\r\n# define AO_HAVE_compare_and_swap\r\n#endif\r\n#if !defined(AO_HAVE_compare_and_swap) && \\\r\n defined(AO_HAVE_compare_and_swap_write)\r\n# define AO_compare_and_swap(addr, old, new_val) \\\r\n AO_compare_and_swap_write(addr, old, new_val)\r\n# define AO_HAVE_compare_and_swap\r\n#endif\r\n#if !defined(AO_HAVE_compare_and_swap) && \\\r\n defined(AO_HAVE_compare_and_swap_read)\r\n# define AO_compare_and_swap(addr, old, new_val) \\\r\n AO_compare_and_swap_read(addr, old, new_val)\r\n# define AO_HAVE_compare_and_swap\r\n#endif\r\n\r\n#if defined(AO_HAVE_compare_and_swap_acquire) &&\\\r\n defined(AO_HAVE_nop_full) && \\\r\n !defined(AO_HAVE_compare_and_swap_full)\r\n# define AO_compare_and_swap_full(addr, old, new_val) \\\r\n (AO_nop_full(), AO_compare_and_swap_acquire(addr, old, new_val))\r\n# define AO_HAVE_compare_and_swap_full\r\n#endif\r\n\r\n#if !defined(AO_HAVE_compare_and_swap_release_write) && \\\r\n defined(AO_HAVE_compare_and_swap_write)\r\n# define AO_compare_and_swap_release_write(addr, old, new_val) \\\r\n AO_compare_and_swap_write(addr, old, new_val)\r\n# define AO_HAVE_compare_and_swap_release_write\r\n#endif\r\n#if !defined(AO_HAVE_compare_and_swap_release_write) && \\\r\n defined(AO_HAVE_compare_and_swap_release)\r\n# define AO_compare_and_swap_release_write(addr, old, new_val) \\\r\n AO_compare_and_swap_release(addr, old, new_val)\r\n# define AO_HAVE_compare_and_swap_release_write\r\n#endif\r\n#if !defined(AO_HAVE_compare_and_swap_acquire_read) && \\\r\n defined(AO_HAVE_compare_and_swap_read)\r\n# define AO_compare_and_swap_acquire_read(addr, old, new_val) \\\r\n AO_compare_and_swap_read(addr, old, new_val)\r\n# define AO_HAVE_compare_and_swap_acquire_read\r\n#endif\r\n#if !defined(AO_HAVE_compare_and_swap_acquire_read) && \\\r\n defined(AO_HAVE_compare_and_swap_acquire)\r\n# define AO_compare_and_swap_acquire_read(addr, old, new_val) \\\r\n AO_compare_and_swap_acquire(addr, old, new_val)\r\n# define AO_HAVE_compare_and_swap_acquire_read\r\n#endif\r\n\r\n#ifdef AO_NO_DD_ORDERING\r\n# if defined(AO_HAVE_compare_and_swap_acquire_read)\r\n# define AO_compare_and_swap_dd_acquire_read(addr, old, new_val) \\\r\n AO_compare_and_swap_acquire_read(addr, old, new_val)\r\n# define AO_HAVE_compare_and_swap_dd_acquire_read\r\n# endif\r\n#else\r\n# if defined(AO_HAVE_compare_and_swap)\r\n# define AO_compare_and_swap_dd_acquire_read(addr, old, new_val) \\\r\n AO_compare_and_swap(addr, old, new_val)\r\n# define AO_HAVE_compare_and_swap_dd_acquire_read\r\n# endif\r\n#endif\r\n\r\n#include \"generalize-small.h\"\r\n\r\n/* Compare_double_and_swap_double */\r\n#if defined(AO_HAVE_compare_double_and_swap_double) && defined(AO_HAVE_nop_full)\\\r\n && !defined(AO_HAVE_compare_double_and_swap_double_acquire)\r\n AO_INLINE int\r\n AO_compare_double_and_swap_double_acquire(volatile AO_double_t *addr,\r\n AO_t o1, AO_t o2,\r\n AO_t n1, AO_t n2)\r\n {\r\n int result = AO_compare_double_and_swap_double(addr, o1, o2, n1, n2);\r\n AO_nop_full();\r\n return result;\r\n }\r\n# define AO_HAVE_compare_double_and_swap_double_acquire\r\n#endif\r\n#if defined(AO_HAVE_compare_double_and_swap_double) \\\r\n && defined(AO_HAVE_nop_full)\\\r\n && !defined(AO_HAVE_compare_double_and_swap_double_release)\r\n# define AO_compare_double_and_swap_double_release(addr, o1, o2, n1, n2) \\\r\n (AO_nop_full(), AO_compare_double_and_swap_double(addr, o1, o2, n1, n2))\r\n# define AO_HAVE_compare_double_and_swap_double_release\r\n#endif\r\n#if defined(AO_HAVE_compare_double_and_swap_double_full)\r\n# if !defined(AO_HAVE_compare_double_and_swap_double_release)\r\n# define AO_compare_double_and_swap_double_release(addr, o1, o2, n1, n2) \\\r\n AO_compare_double_and_swap_double_full(addr, o1, o2, n1, n2)\r\n# define AO_HAVE_compare_double_and_swap_double_release\r\n# endif\r\n# if !defined(AO_HAVE_compare_double_and_swap_double_acquire)\r\n# define AO_compare_double_and_swap_double_acquire(addr, o1, o2, n1, n2) \\\r\n AO_compare_double_and_swap_double_full(addr, o1, o2, n1, n2)\r\n# define AO_HAVE_compare_double_and_swap_double_acquire\r\n# endif\r\n# if !defined(AO_HAVE_compare_double_and_swap_double_write)\r\n# define AO_compare_double_and_swap_double_write(addr, o1, o2, n1, n2) \\\r\n AO_compare_double_and_swap_double_full(addr, o1, o2, n1, n2)\r\n# define AO_HAVE_compare_double_and_swap_double_write\r\n# endif\r\n# if !defined(AO_HAVE_compare_double_and_swap_double_read)\r\n# define AO_compare_double_and_swap_double_read(addr, o1, o2, n1, n2) \\\r\n AO_compare_double_and_swap_double_full(addr, o1, o2, n1, n2)\r\n# define AO_HAVE_compare_double_and_swap_double_read\r\n# endif\r\n#endif /* AO_HAVE_compare_double_and_swap_double_full */\r\n\r\n#if !defined(AO_HAVE_compare_double_and_swap_double) && \\\r\n defined(AO_HAVE_compare_double_and_swap_double_release)\r\n# define AO_compare_double_and_swap_double(addr, o1, o2, n1, n2) \\\r\n AO_compare_double_and_swap_double_release(addr, o1, o2, n1, n2)\r\n# define AO_HAVE_compare_double_and_swap_double\r\n#endif\r\n#if !defined(AO_HAVE_compare_double_and_swap_double) && \\\r\n defined(AO_HAVE_compare_double_and_swap_double_acquire)\r\n# define AO_compare_double_and_swap_double(addr, o1, o2, n1, n2) \\\r\n AO_compare_double_and_swap_double_acquire(addr, o1, o2, n1, n2)\r\n# define AO_HAVE_compare_double_and_swap_double\r\n#endif\r\n#if !defined(AO_HAVE_compare_double_and_swap_double) && \\\r\n defined(AO_HAVE_compare_double_and_swap_double_write)\r\n# define AO_compare_double_and_swap_double(addr, o1, o2, n1, n2) \\\r\n AO_compare_double_and_swap_double_write(addr, o1, o2, n1, n2)\r\n# define AO_HAVE_compare_double_and_swap_double\r\n#endif\r\n#if !defined(AO_HAVE_compare_double_and_swap_double) && \\\r\n defined(AO_HAVE_compare_double_and_swap_double_read)\r\n# define AO_compare_double_and_swap_double(addr, o1, o2, n1, n2) \\\r\n AO_compare_double_and_swap_double_read(addr, o1, o2, n1, n2)\r\n# define AO_HAVE_compare_double_and_swap_double\r\n#endif\r\n\r\n#if defined(AO_HAVE_compare_double_and_swap_double_acquire) &&\\\r\n defined(AO_HAVE_nop_full) && \\\r\n !defined(AO_HAVE_compare_double_and_swap_double_full)\r\n# define AO_compare_double_and_swap_double_full(addr, o1, o2, n1, n2) \\\r\n (AO_nop_full(), AO_compare_double_and_swap_double_acquire(addr, o1, o2, n1, n2))\r\n# define AO_HAVE_compare_double_and_swap_double_full\r\n#endif\r\n\r\n#if !defined(AO_HAVE_compare_double_and_swap_double_release_write) && \\\r\n defined(AO_HAVE_compare_double_and_swap_double_write)\r\n# define AO_compare_double_and_swap_double_release_write(addr, o1, o2, n1, n2) \\\r\n AO_compare_double_and_swap_double_write(addr, o1, o2, n1, n2)\r\n# define AO_HAVE_compare_double_and_swap_double_release_write\r\n#endif\r\n#if !defined(AO_HAVE_compare_double_and_swap_double_release_write) && \\\r\n defined(AO_HAVE_compare_double_and_swap_double_release)\r\n# define AO_compare_double_and_swap_double_release_write(addr, o1, o2, n1, n2) \\\r\n AO_compare_double_and_swap_double_release(addr, o1, o2, n1, n2)\r\n# define AO_HAVE_compare_double_and_swap_double_release_write\r\n#endif\r\n#if !defined(AO_HAVE_compare_double_and_swap_double_acquire_read) && \\\r\n defined(AO_HAVE_compare_double_and_swap_double_read)\r\n# define AO_compare_double_and_swap_double_acquire_read(addr, o1, o2, n1, n2) \\\r\n AO_compare_double_and_swap_double_read(addr, o1, o2, n1, n2)\r\n# define AO_HAVE_compare_double_and_swap_double_acquire_read\r\n#endif\r\n#if !defined(AO_HAVE_compare_double_and_swap_double_acquire_read) && \\\r\n defined(AO_HAVE_compare_double_and_swap_double_acquire)\r\n# define AO_compare_double_and_swap_double_acquire_read(addr, o1, o2, n1, n2) \\\r\n AO_compare_double_and_swap_double_acquire(addr, o1, o2, n1, n2)\r\n# define AO_HAVE_compare_double_and_swap_double_acquire_read\r\n#endif\r\n\r\n#ifdef AO_NO_DD_ORDERING\r\n# if defined(AO_HAVE_compare_double_and_swap_double_acquire_read)\r\n# define AO_compare_double_and_swap_double_dd_acquire_read(addr, o1, o2, n1, n2) \\\r\n AO_compare_double_and_swap_double_acquire_read(addr, o1, o2, n1, n2)\r\n# define AO_HAVE_compare_double_and_swap_double_dd_acquire_read\r\n# endif\r\n#else\r\n# if defined(AO_HAVE_compare_double_and_swap_double)\r\n# define AO_compare_double_and_swap_double_dd_acquire_read(addr, o1, o2, n1, n2) \\\r\n AO_compare_double_and_swap_double(addr, o1, o2, n1, n2)\r\n# define AO_HAVE_compare_double_and_swap_double_dd_acquire_read\r\n# endif\r\n#endif\r\n\r\n/* Compare_and_swap_double */\r\n#if defined(AO_HAVE_compare_and_swap_double) && defined(AO_HAVE_nop_full)\\\r\n && !defined(AO_HAVE_compare_and_swap_double_acquire)\r\n AO_INLINE int\r\n AO_compare_and_swap_double_acquire(volatile AO_double_t *addr,\r\n AO_t o1,\r\n AO_t n1, AO_t n2)\r\n {\r\n int result = AO_compare_and_swap_double(addr, o1, n1, n2);\r\n AO_nop_full();\r\n return result;\r\n }\r\n# define AO_HAVE_compare_and_swap_double_acquire\r\n#endif\r\n#if defined(AO_HAVE_compare_and_swap_double) \\\r\n && defined(AO_HAVE_nop_full)\\\r\n && !defined(AO_HAVE_compare_and_swap_double_release)\r\n# define AO_compare_and_swap_double_release(addr, o1, n1, n2) \\\r\n (AO_nop_full(), AO_compare_and_swap_double(addr, o1, n1, n2))\r\n# define AO_HAVE_compare_and_swap_double_release\r\n#endif\r\n#if defined(AO_HAVE_compare_and_swap_double_full)\r\n# if !defined(AO_HAVE_compare_and_swap_double_release)\r\n# define AO_compare_and_swap_double_release(addr, o1, n1, n2) \\\r\n AO_compare_and_swap_double_full(addr, o1, n1, n2)\r\n# define AO_HAVE_compare_and_swap_double_release\r\n# endif\r\n# if !defined(AO_HAVE_compare_and_swap_double_acquire)\r\n# define AO_compare_and_swap_double_acquire(addr, o1, n1, n2) \\\r\n AO_compare_and_swap_double_full(addr, o1, n1, n2)\r\n# define AO_HAVE_compare_and_swap_double_acquire\r\n# endif\r\n# if !defined(AO_HAVE_compare_and_swap_double_write)\r\n# define AO_compare_and_swap_double_write(addr, o1, n1, n2) \\\r\n AO_compare_and_swap_double_full(addr, o1, n1, n2)\r\n# define AO_HAVE_compare_and_swap_double_write\r\n# endif\r\n# if !defined(AO_HAVE_compare_and_swap_double_read)\r\n# define AO_compare_and_swap_double_read(addr, o1, n1, n2) \\\r\n AO_compare_and_swap_double_full(addr, o1, n1, n2)\r\n# define AO_HAVE_compare_and_swap_double_read\r\n# endif\r\n#endif /* AO_HAVE_compare_and_swap_double_full */\r\n\r\n#if !defined(AO_HAVE_compare_and_swap_double) && \\\r\n defined(AO_HAVE_compare_and_swap_double_release)\r\n# define AO_compare_and_swap_double(addr, o1, n1, n2) \\\r\n AO_compare_and_swap_double_release(addr, o1, n1, n2)\r\n# define AO_HAVE_compare_and_swap_double\r\n#endif\r\n#if !defined(AO_HAVE_compare_and_swap_double) && \\\r\n defined(AO_HAVE_compare_and_swap_double_acquire)\r\n# define AO_compare_and_swap_double(addr, o1, n1, n2) \\\r\n AO_compare_and_swap_double_acquire(addr, o1, n1, n2)\r\n# define AO_HAVE_compare_and_swap_double\r\n#endif\r\n#if !defined(AO_HAVE_compare_and_swap_double) && \\\r\n defined(AO_HAVE_compare_and_swap_double_write)\r\n# define AO_compare_and_swap_double(addr, o1, n1, n2) \\\r\n AO_compare_and_swap_double_write(addr, o1, n1, n2)\r\n# define AO_HAVE_compare_and_swap_double\r\n#endif\r\n#if !defined(AO_HAVE_compare_and_swap_double) && \\\r\n defined(AO_HAVE_compare_and_swap_double_read)\r\n# define AO_compare_and_swap_double(addr, o1, n1, n2) \\\r\n AO_compare_and_swap_double_read(addr, o1, n1, n2)\r\n# define AO_HAVE_compare_and_swap_double\r\n#endif\r\n\r\n#if defined(AO_HAVE_compare_and_swap_double_acquire) &&\\\r\n defined(AO_HAVE_nop_full) && \\\r\n !defined(AO_HAVE_compare_and_swap_double_full)\r\n# define AO_compare_and_swap_double_full(addr, o1, n1, n2) \\\r\n (AO_nop_full(), AO_compare_and_swap_double_acquire(addr, o1, n1, n2))\r\n# define AO_HAVE_compare_and_swap_double_full\r\n#endif\r\n\r\n#if !defined(AO_HAVE_compare_and_swap_double_release_write) && \\\r\n defined(AO_HAVE_compare_and_swap_double_write)\r\n# define AO_compare_and_swap_double_release_write(addr, o1, n1, n2) \\\r\n AO_compare_and_swap_double_write(addr, o1, n1, n2)\r\n# define AO_HAVE_compare_and_swap_double_release_write\r\n#endif\r\n#if !defined(AO_HAVE_compare_and_swap_double_release_write) && \\\r\n defined(AO_HAVE_compare_and_swap_double_release)\r\n# define AO_compare_and_swap_double_release_write(addr, o1, n1, n2) \\\r\n AO_compare_and_swap_double_release(addr, o1, n1, n2)\r\n# define AO_HAVE_compare_and_swap_double_release_write\r\n#endif\r\n#if !defined(AO_HAVE_compare_and_swap_double_acquire_read) && \\\r\n defined(AO_HAVE_compare_and_swap_double_read)\r\n# define AO_compare_and_swap_double_acquire_read(addr, o1, n1, n2) \\\r\n AO_compare_and_swap_double_read(addr, o1, n1, n2)\r\n# define AO_HAVE_compare_and_swap_double_acquire_read\r\n#endif\r\n#if !defined(AO_HAVE_compare_and_swap_double_acquire_read) && \\\r\n defined(AO_HAVE_compare_and_swap_double_acquire)\r\n# define AO_compare_and_swap_double_acquire_read(addr, o1, n1, n2) \\\r\n AO_compare_and_swap_double_acquire(addr, o1, n1, n2)\r\n# define AO_HAVE_compare_and_swap_double_acquire_read\r\n#endif\r\n\r\n#ifdef AO_NO_DD_ORDERING\r\n# if defined(AO_HAVE_compare_and_swap_double_acquire_read)\r\n# define AO_compare_and_swap_double_dd_acquire_read(addr, o1, n1, n2) \\\r\n AO_compare_and_swap_double_acquire_read(addr, o1, n1, n2)\r\n# define AO_HAVE_compare_and_swap_double_dd_acquire_read\r\n# endif\r\n#else\r\n# if defined(AO_HAVE_compare_and_swap_double)\r\n# define AO_compare_and_swap_double_dd_acquire_read(addr, o1, n1, n2) \\\r\n AO_compare_and_swap_double(addr, o1, n1, n2)\r\n# define AO_HAVE_compare_and_swap_double_dd_acquire_read\r\n# endif\r\n#endif\r\n\r\n/* NEC LE-IT: Convenience functions for AO_double compare and swap which */\r\n/* types and reads easier in code */\r\n#if defined(AO_HAVE_compare_double_and_swap_double_release) && \\\r\n !defined(AO_HAVE_double_compare_and_swap_release)\r\nAO_INLINE int\r\nAO_double_compare_and_swap_release(volatile AO_double_t *addr,\r\n AO_double_t old_val, AO_double_t new_val)\r\n{\r\n return AO_compare_double_and_swap_double_release(addr,\r\n old_val.AO_val1, old_val.AO_val2,\r\n new_val.AO_val1, new_val.AO_val2);\r\n}\r\n#define AO_HAVE_double_compare_and_swap_release\r\n#endif\r\n\r\n#if defined(AO_HAVE_compare_double_and_swap_double_acquire) && \\\r\n !defined(AO_HAVE_double_compare_and_swap_acquire)\r\nAO_INLINE int\r\nAO_double_compare_and_swap_acquire(volatile AO_double_t *addr,\r\n AO_double_t old_val, AO_double_t new_val)\r\n{\r\n return AO_compare_double_and_swap_double_acquire(addr,\r\n old_val.AO_val1, old_val.AO_val2,\r\n new_val.AO_val1, new_val.AO_val2);\r\n}\r\n#define AO_HAVE_double_compare_and_swap_acquire\r\n#endif\r\n\r\n#if defined(AO_HAVE_compare_double_and_swap_double_full) && \\\r\n !defined(AO_HAVE_double_compare_and_swap_full)\r\nAO_INLINE int\r\nAO_double_compare_and_swap_full(volatile AO_double_t *addr,\r\n AO_double_t old_val, AO_double_t new_val)\r\n{\r\n return AO_compare_double_and_swap_double_full(addr,\r\n old_val.AO_val1, old_val.AO_val2,\r\n new_val.AO_val1, new_val.AO_val2);\r\n}\r\n#define AO_HAVE_double_compare_and_swap_full\r\n#endif\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/README", "content": "There are two kinds of entities in this directory:\r\n\r\n- Subdirectories corresponding to specific compilers (or compiler/OS combinations).\r\n Each of these includes one or more architecture-specific headers.\r\n\r\n- More generic header files corresponding to a particular ordering and/or\r\n atomicity property that might be shared by multiple hardware platforms.\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/acquire_release_volatile.h", "content": "/*\r\n * Copyright (c) 2003 Hewlett-Packard Development Company, L.P.\r\n *\r\n * Permission is hereby granted, free of charge, to any person obtaining a copy\r\n * of this software and associated documentation files (the \"Software\"), to deal\r\n * in the Software without restriction, including without limitation the rights\r\n * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\r\n * copies of the Software, and to permit persons to whom the Software is\r\n * furnished to do so, subject to the following conditions:\r\n *\r\n * The above copyright notice and this permission notice shall be included in\r\n * all copies or substantial portions of the Software.\r\n *\r\n * THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\r\n * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\r\n * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\r\n * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\r\n * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\r\n * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\r\n * SOFTWARE.\r\n */\r\n\r\n/*\r\n * This file adds definitions appropriate for environments in which an AO_t\r\n * volatile load has acquire semantics, and an AO_t volatile store has release\r\n * semantics. This is arguably supposed to be true with the standard Itanium\r\n * software conventions.\r\n */\r\n\r\n/*\r\n * Empirically gcc/ia64 does some reordering of ordinary operations around volatiles\r\n * even when we think it shouldn't. Gcc 3.3 and earlier could reorder a volatile store\r\n * with another store. As of March 2005, gcc pre-4 reused previously computed\r\n * common subexpressions across a volatile load.\r\n * Hence we now add compiler barriers for gcc.\r\n */\r\n#if !defined(AO_GCC_BARRIER)\r\n# if defined(__GNUC__)\r\n# define AO_GCC_BARRIER() AO_compiler_barrier()\r\n# else\r\n# define AO_GCC_BARRIER()\r\n# endif\r\n#endif\r\n\r\nAO_INLINE AO_t\r\nAO_load_acquire(const volatile AO_t *p)\r\n{\r\n AO_t result = *p;\r\n /* A normal volatile load generates an ld.acq */\r\n AO_GCC_BARRIER();\r\n return result;\r\n}\r\n#define AO_HAVE_load_acquire\r\n\r\nAO_INLINE void\r\nAO_store_release(volatile AO_t *p, AO_t val)\r\n{\r\n AO_GCC_BARRIER();\r\n /* A normal volatile store generates an st.rel */\r\n *p = val;\r\n}\r\n#define AO_HAVE_store_release\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/aligned_atomic_load_store.h", "content": "/*\r\n * Copyright (c) 2003 Hewlett-Packard Development Company, L.P.\r\n *\r\n * Permission is hereby granted, free of charge, to any person obtaining a copy\r\n * of this software and associated documentation files (the \"Software\"), to deal\r\n * in the Software without restriction, including without limitation the rights\r\n * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\r\n * copies of the Software, and to permit persons to whom the Software is\r\n * furnished to do so, subject to the following conditions:\r\n *\r\n * The above copyright notice and this permission notice shall be included in\r\n * all copies or substantial portions of the Software.\r\n *\r\n * THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\r\n * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\r\n * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\r\n * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\r\n * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\r\n * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\r\n * SOFTWARE.\r\n */\r\n\r\n/*\r\n * Definitions for architectures on which loads and stores of AO_t are\r\n * atomic fo all legal alignments.\r\n */\r\n\r\nAO_INLINE AO_t\r\nAO_load(const volatile AO_t *addr)\r\n{\r\n assert(((size_t)addr & (sizeof(AO_t) - 1)) == 0);\r\n /* Cast away the volatile for architectures where */\r\n /* volatile adds barrier semantics. */\r\n return *(AO_t *)addr;\r\n}\r\n\r\n#define AO_HAVE_load\r\n\r\nAO_INLINE void\r\nAO_store(volatile AO_t *addr, AO_t new_val)\r\n{\r\n assert(((size_t)addr & (sizeof(AO_t) - 1)) == 0);\r\n (*(AO_t *)addr) = new_val;\r\n}\r\n\r\n#define AO_HAVE_store\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/all_acquire_release_volatile.h", "content": "/*\r\n * Copyright (c) 2004 Hewlett-Packard Development Company, L.P.\r\n * \r\n * Permission is hereby granted, free of charge, to any person obtaining a copy\r\n * of this software and associated documentation files (the \"Software\"), to deal\r\n * in the Software without restriction, including without limitation the rights\r\n * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\r\n * copies of the Software, and to permit persons to whom the Software is\r\n * furnished to do so, subject to the following conditions:\r\n * \r\n * The above copyright notice and this permission notice shall be included in\r\n * all copies or substantial portions of the Software.\r\n * \r\n * THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\r\n * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\r\n * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\r\n * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\r\n * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\r\n * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\r\n * SOFTWARE. \r\n */\r\n\r\n/*\r\n * Describes architectures on which volatile AO_t, unsigned char, unsigned\r\n * short, and unsigned int loads and stores have acquire/release semantics for\r\n * all normally legal alignments.\r\n */\r\n#include \"acquire_release_volatile.h\"\r\n#include \"char_acquire_release_volatile.h\"\r\n#include \"short_acquire_release_volatile.h\"\r\n#include \"int_acquire_release_volatile.h\"\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/all_aligned_atomic_load_store.h", "content": "/*\r\n * Copyright (c) 2004 Hewlett-Packard Development Company, L.P.\r\n *\r\n * Permission is hereby granted, free of charge, to any person obtaining a copy\r\n * of this software and associated documentation files (the \"Software\"), to deal\r\n * in the Software without restriction, including without limitation the rights\r\n * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\r\n * copies of the Software, and to permit persons to whom the Software is\r\n * furnished to do so, subject to the following conditions:\r\n *\r\n * The above copyright notice and this permission notice shall be included in\r\n * all copies or substantial portions of the Software.\r\n *\r\n * THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\r\n * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\r\n * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\r\n * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\r\n * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\r\n * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\r\n * SOFTWARE.\r\n */\r\n\r\n/*\r\n * Describes architectures on which AO_t, unsigned char, unsigned short,\r\n * and unsigned int loads and stores are atomic for all normally legal\r\n * alignments.\r\n */\r\n#include \"aligned_atomic_load_store.h\"\r\n#include \"char_atomic_load_store.h\"\r\n#include \"short_aligned_atomic_load_store.h\"\r\n#include \"int_aligned_atomic_load_store.h\"\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/all_atomic_load_store.h", "content": "/*\r\n * Copyright (c) 2004 Hewlett-Packard Development Company, L.P.\r\n *\r\n * Permission is hereby granted, free of charge, to any person obtaining a copy\r\n * of this software and associated documentation files (the \"Software\"), to deal\r\n * in the Software without restriction, including without limitation the rights\r\n * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\r\n * copies of the Software, and to permit persons to whom the Software is\r\n * furnished to do so, subject to the following conditions:\r\n *\r\n * The above copyright notice and this permission notice shall be included in\r\n * all copies or substantial portions of the Software.\r\n *\r\n * THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\r\n * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\r\n * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\r\n * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\r\n * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\r\n * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\r\n * SOFTWARE.\r\n */\r\n\r\n/*\r\n * Describes architectures on which AO_t, unsigned char, unsigned short,\r\n * and unsigned int loads and stores are atomic for all normally legal\r\n * alignments.\r\n */\r\n#include \"atomic_load_store.h\"\r\n#include \"char_atomic_load_store.h\"\r\n#include \"short_atomic_load_store.h\"\r\n#include \"int_atomic_load_store.h\"\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/ao_t_is_int.h", "content": "/*\r\n * Copyright (c) 2003-2004 Hewlett-Packard Development Company, L.P.\r\n *\r\n * Permission is hereby granted, free of charge, to any person obtaining a copy\r\n * of this software and associated documentation files (the \"Software\"), to deal\r\n * in the Software without restriction, including without limitation the rights\r\n * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\r\n * copies of the Software, and to permit persons to whom the Software is\r\n * furnished to do so, subject to the following conditions:\r\n *\r\n * The above copyright notice and this permission notice shall be included in\r\n * all copies or substantial portions of the Software.\r\n *\r\n * THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\r\n * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\r\n * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\r\n * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\r\n * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\r\n * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\r\n * SOFTWARE.\r\n */\r\n\r\n/*\r\n * Inclusion of this file signifies that AO_t is in fact int. Hence\r\n * any AO_... operations can also server as AO_int_... operations.\r\n * We currently define only the more important ones here, and allow for\r\n * the normal generalization process to define the others.\r\n * We should probably add others in the future.\r\n */\r\n\r\n#if defined(AO_HAVE_compare_and_swap_full) && \\\r\n !defined(AO_HAVE_int_compare_and_swap_full)\r\n# define AO_int_compare_and_swap_full(addr, old, new_val) \\\r\n AO_compare_and_swap_full((volatile AO_t *)(addr), \\\r\n (AO_t)(old), (AO_t)(new_val))\r\n# define AO_HAVE_int_compare_and_swap_full\r\n# endif\r\n\r\n#if defined(AO_HAVE_compare_and_swap_acquire) && \\\r\n !defined(AO_HAVE_int_compare_and_swap_acquire)\r\n# define AO_int_compare_and_swap_acquire(addr, old, new_val) \\\r\n AO_compare_and_swap_acquire((volatile AO_t *)(addr), \\\r\n (AO_t)(old), (AO_t)(new_val))\r\n# define AO_HAVE_int_compare_and_swap_acquire\r\n# endif\r\n\r\n#if defined(AO_HAVE_compare_and_swap_release) && \\\r\n !defined(AO_HAVE_int_compare_and_swap_release)\r\n# define AO_int_compare_and_swap_release(addr, old, new_val) \\\r\n AO_compare_and_swap_release((volatile AO_t *)(addr), \\\r\n (AO_t)(old), (AO_t)(new_val))\r\n# define AO_HAVE_int_compare_and_swap_release\r\n# endif\r\n\r\n#if defined(AO_HAVE_compare_and_swap_write) && \\\r\n !defined(AO_HAVE_int_compare_and_swap_write)\r\n# define AO_int_compare_and_swap_write(addr, old, new_val) \\\r\n AO_compare_and_swap_write((volatile AO_t *)(addr), \\\r\n (AO_t)(old), (AO_t)(new_val))\r\n# define AO_HAVE_int_compare_and_swap_write\r\n# endif\r\n\r\n#if defined(AO_HAVE_compare_and_swap_read) && \\\r\n !defined(AO_HAVE_int_compare_and_swap_read)\r\n# define AO_int_compare_and_swap_read(addr, old, new_val) \\\r\n AO_compare_and_swap_read((volatile AO_t *)(addr), \\\r\n (AO_t)(old), (AO_t)(new_val))\r\n# define AO_HAVE_int_compare_and_swap_read\r\n# endif\r\n\r\n#if defined(AO_HAVE_compare_and_swap) && \\\r\n !defined(AO_HAVE_int_compare_and_swap)\r\n# define AO_int_compare_and_swap(addr, old, new_val) \\\r\n AO_compare_and_swap((volatile AO_t *)(addr), \\\r\n (AO_t)(old), (AO_t)(new_val))\r\n# define AO_HAVE_int_compare_and_swap\r\n# endif\r\n\r\n#if defined(AO_HAVE_load_acquire) && \\\r\n !defined(AO_HAVE_int_load_acquire)\r\n# define AO_int_load_acquire(addr) \\\r\n (int)AO_load_acquire((const volatile AO_t *)(addr))\r\n# define AO_HAVE_int_load_acquire\r\n# endif\r\n\r\n#if defined(AO_HAVE_store_release) && \\\r\n !defined(AO_HAVE_int_store_release)\r\n# define AO_int_store_release(addr, val) \\\r\n AO_store_release((volatile AO_t *)(addr), (AO_t)(val))\r\n# define AO_HAVE_int_store_release\r\n# endif\r\n\r\n#if defined(AO_HAVE_fetch_and_add_full) && \\\r\n !defined(AO_HAVE_int_fetch_and_add_full)\r\n# define AO_int_fetch_and_add_full(addr, incr) \\\r\n (int)AO_fetch_and_add_full((volatile AO_t *)(addr), (AO_t)(incr))\r\n# define AO_HAVE_int_fetch_and_add_full\r\n# endif\r\n\r\n#if defined(AO_HAVE_fetch_and_add1_acquire) && \\\r\n !defined(AO_HAVE_int_fetch_and_add1_acquire)\r\n# define AO_int_fetch_and_add1_acquire(addr) \\\r\n (int)AO_fetch_and_add1_acquire((volatile AO_t *)(addr))\r\n# define AO_HAVE_int_fetch_and_add1_acquire\r\n# endif\r\n\r\n#if defined(AO_HAVE_fetch_and_add1_release) && \\\r\n !defined(AO_HAVE_int_fetch_and_add1_release)\r\n# define AO_int_fetch_and_add1_release(addr) \\\r\n (int)AO_fetch_and_add1_release((volatile AO_t *)(addr))\r\n# define AO_HAVE_int_fetch_and_add1_release\r\n# endif\r\n\r\n#if defined(AO_HAVE_fetch_and_sub1_acquire) && \\\r\n !defined(AO_HAVE_int_fetch_and_sub1_acquire)\r\n# define AO_int_fetch_and_sub1_acquire(addr) \\\r\n (int)AO_fetch_and_sub1_acquire((volatile AO_t *)(addr))\r\n# define AO_HAVE_int_fetch_and_sub1_acquire\r\n# endif\r\n\r\n#if defined(AO_HAVE_fetch_and_sub1_release) && \\\r\n !defined(AO_HAVE_int_fetch_and_sub1_release)\r\n# define AO_int_fetch_and_sub1_release(addr) \\\r\n (int)AO_fetch_and_sub1_release((volatile AO_t *)(addr))\r\n# define AO_HAVE_int_fetch_and_sub1_release\r\n# endif\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/armcc/arm_v6.h", "content": "/*\r\n * Copyright (c) 2007 by NEC LE-IT: All rights reserved.\r\n * A transcription of ARMv6 atomic operations for the ARM Realview Toolchain.\r\n * This code works with armcc from RVDS 3.1\r\n * This is based on work in gcc/arm.h by\r\n * Copyright (c) 1991-1994 by Xerox Corporation. All rights reserved.\r\n * Copyright (c) 1996-1999 by Silicon Graphics. All rights reserved.\r\n * Copyright (c) 1999-2003 by Hewlett-Packard Company. All rights reserved.\r\n *\r\n *\r\n *\r\n * THIS MATERIAL IS PROVIDED AS IS, WITH ABSOLUTELY NO WARRANTY EXPRESSED\r\n * OR IMPLIED. ANY USE IS AT YOUR OWN RISK.\r\n *\r\n * Permission is hereby granted to use or copy this program\r\n * for any purpose, provided the above notices are retained on all copies.\r\n * Permission to modify the code and to distribute modified code is granted,\r\n * provided the above notices are retained, and a notice that the code was\r\n * modified is included with the above copyright notice.\r\n *\r\n */\r\n#include \"../read_ordered.h\"\r\n#include \"../test_and_set_t_is_ao_t.h\" /* Probably suboptimal */\r\n\r\n#if __TARGET_ARCH_ARM < 6\r\nDont use with ARM instruction sets lower than v6\r\n#else\r\n\r\n#include \"../standard_ao_double_t.h\"\r\n\r\n/* NEC LE-IT: ARMv6 is the first architecture providing support for simple LL/SC\r\n * A data memory barrier must be raised via CP15 command (see documentation).\r\n *\r\n * ARMv7 is compatible to ARMv6 but has a simpler command for issuing a\r\n * memory barrier (DMB). Raising it via CP15 should still work as told me by the\r\n * support engineers. If it turns out to be much quicker than we should implement\r\n * custom code for ARMv7 using the asm { dmb } command.\r\n *\r\n * If only a single processor is used, we can define AO_UNIPROCESSOR\r\n * and do not need to access CP15 for ensuring a DMB at all.\r\n*/\r\n\r\nAO_INLINE void\r\nAO_nop_full(void)\r\n{\r\n#ifndef AO_UNIPROCESSOR\r\n unsigned int dest=0;\r\n /* issue an data memory barrier (keeps ordering of memory transactions */\r\n /* before and after this operation) */\r\n __asm { mcr p15,0,dest,c7,c10,5 } ;\r\n#endif\r\n}\r\n\r\n#define AO_HAVE_nop_full\r\n\r\nAO_INLINE AO_t\r\nAO_load(const volatile AO_t *addr)\r\n{\r\n /* Cast away the volatile in case it adds fence semantics */\r\n return (*(const AO_t *)addr);\r\n}\r\n#define AO_HAVE_load\r\n\r\n/* NEC LE-IT: atomic \"store\" - according to ARM documentation this is\r\n * the only safe way to set variables also used in LL/SC environment.\r\n * A direct write won't be recognized by the LL/SC construct in other CPUs.\r\n *\r\n * HB: Based on subsequent discussion, I think it would be OK to use an\r\n * ordinary store here if we knew that interrupt handlers always cleared\r\n * the reservation. They should, but there is some doubt that this is\r\n * currently always the case for e.g. Linux.\r\n*/\r\nAO_INLINE void AO_store(volatile AO_t *addr, AO_t value)\r\n{\r\n unsigned long tmp;\r\n\r\nretry:\r\n__asm {\r\n ldrex tmp, [addr]\r\n strex tmp, value, [addr]\r\n teq tmp, #0\r\n bne retry\r\n };\r\n}\r\n#define AO_HAVE_store\r\n\r\n/* NEC LE-IT: replace the SWAP as recommended by ARM:\r\n\r\n \"Applies to: ARM11 Cores\r\n Though the SWP instruction will still work with ARM V6 cores, it is recommended\r\n to use the new V6 synchronization instructions. The SWP instruction produces\r\n locked read and write accesses which are atomic, i.e. another operation cannot\r\n be done between these locked accesses which ties up external bus (AHB,AXI)\r\n bandwidth and can increase worst case interrupt latencies. LDREX,STREX are\r\n more flexible, other instructions can be done between the LDREX and STREX accesses.\r\n \"\r\n*/\r\nAO_INLINE AO_TS_t\r\nAO_test_and_set(volatile AO_TS_t *addr) {\r\n\r\n AO_TS_t oldval;\r\n unsigned long tmp;\r\n unsigned long one = 1;\r\nretry:\r\n__asm {\r\n ldrex oldval, [addr]\r\n strex tmp, one, [addr]\r\n teq tmp, #0\r\n bne retry\r\n }\r\n\r\n return oldval;\r\n}\r\n\r\n#define AO_HAVE_test_and_set\r\n\r\n/* NEC LE-IT: fetch and add for ARMv6 */\r\nAO_INLINE AO_t\r\nAO_fetch_and_add(volatile AO_t *p, AO_t incr)\r\n{\r\n unsigned long tmp,tmp2;\r\n AO_t result;\r\n\r\nretry:\r\n__asm {\r\n ldrex result, [p]\r\n add tmp, incr, result\r\n strex tmp2, tmp, [p]\r\n teq tmp2, #0\r\n bne retry }\r\n\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_fetch_and_add\r\n\r\n/* NEC LE-IT: fetch and add1 for ARMv6 */\r\nAO_INLINE AO_t\r\nAO_fetch_and_add1(volatile AO_t *p)\r\n{\r\n unsigned long tmp,tmp2;\r\n AO_t result;\r\n\r\nretry:\r\n__asm {\r\n ldrex result, [p]\r\n add tmp, result, #1\r\n strex tmp2, tmp, [p]\r\n teq tmp2, #0\r\n bne retry\r\n }\r\n\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_fetch_and_add1\r\n\r\n/* NEC LE-IT: fetch and sub for ARMv6 */\r\nAO_INLINE AO_t\r\nAO_fetch_and_sub1(volatile AO_t *p)\r\n{\r\n unsigned long tmp,tmp2;\r\n AO_t result;\r\n\r\nretry:\r\n__asm {\r\n ldrex result, [p]\r\n sub tmp, result, #1\r\n strex tmp2, tmp, [p]\r\n teq tmp2, #0\r\n bne retry\r\n }\r\n\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_fetch_and_sub1\r\n\r\n/* NEC LE-IT: compare and swap */\r\n/* Returns nonzero if the comparison succeeded. */\r\nAO_INLINE int\r\nAO_compare_and_swap(volatile AO_t *addr,\r\n AO_t old_val, AO_t new_val)\r\n{\r\n AO_t result,tmp;\r\n\r\nretry:\r\n__asm__ {\r\n mov result, #2\r\n ldrex tmp, [addr]\r\n teq tmp, old_val\r\n it eq\r\n strexeq result, new_val, [addr]\r\n teq result, #1\r\n beq retry\r\n }\r\n\r\n return !(result&2);\r\n}\r\n#define AO_HAVE_compare_and_swap\r\n\r\n/* helper functions for the Realview compiler: LDREXD is not usable\r\n * with inline assembler, so use the \"embedded\" assembler as\r\n * suggested by ARM Dev. support (June 2008). */\r\n__asm inline double_ptr_storage load_ex(volatile AO_double_t *addr) {\r\n LDREXD r0,r1,[r0]\r\n}\r\n\r\n__asm inline int store_ex(AO_t val1, AO_t val2, volatile AO_double_t *addr) {\r\n STREXD r3,r0,r1,[r2]\r\n MOV r0,r3\r\n}\r\n\r\nAO_INLINE int\r\nAO_compare_double_and_swap_double(volatile AO_double_t *addr,\r\n AO_t old_val1, AO_t old_val2,\r\n AO_t new_val1, AO_t new_val2)\r\n{\r\n double_ptr_storage old_val = ((double_ptr_storage)old_val2 << 32) | old_val1;\r\n\r\n double_ptr_storage tmp;\r\n int result;\r\n\r\n while(1) {\r\n tmp = load_ex(addr);\r\n if(tmp != old_val) return 0;\r\n result = store_ex(new_val1, new_val2, addr);\r\n if(!result) return 1;\r\n }\r\n}\r\n\r\n#define AO_HAVE_compare_double_and_swap_double\r\n\r\n\r\n#endif // __TARGET_ARCH_ARM\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/atomic_load_store.h", "content": "/*\r\n * Copyright (c) 2003 Hewlett-Packard Development Company, L.P.\r\n *\r\n * Permission is hereby granted, free of charge, to any person obtaining a copy\r\n * of this software and associated documentation files (the \"Software\"), to deal\r\n * in the Software without restriction, including without limitation the rights\r\n * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\r\n * copies of the Software, and to permit persons to whom the Software is\r\n * furnished to do so, subject to the following conditions:\r\n *\r\n * The above copyright notice and this permission notice shall be included in\r\n * all copies or substantial portions of the Software.\r\n *\r\n * THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\r\n * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\r\n * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\r\n * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\r\n * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\r\n * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\r\n * SOFTWARE.\r\n */\r\n\r\n/*\r\n * Definitions for architectures on which loads and stores of AO_t are\r\n * atomic for all legal alignments.\r\n */\r\n\r\nAO_INLINE AO_t\r\nAO_load(const volatile AO_t *addr)\r\n{\r\n /* Cast away the volatile for architectures like IA64 where */\r\n /* volatile adds barrier semantics. */\r\n return (*(const AO_t *)addr);\r\n}\r\n\r\n#define AO_HAVE_load\r\n\r\nAO_INLINE void\r\nAO_store(volatile AO_t *addr, AO_t new_val)\r\n{\r\n (*(AO_t *)addr) = new_val;\r\n}\r\n\r\n#define AO_HAVE_store\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/char_acquire_release_volatile.h", "content": "/*\r\n * Copyright (c) 2003-2004 Hewlett-Packard Development Company, L.P.\r\n *\r\n * Permission is hereby granted, free of charge, to any person obtaining a copy\r\n * of this software and associated documentation files (the \"Software\"), to deal\r\n * in the Software without restriction, including without limitation the rights\r\n * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\r\n * copies of the Software, and to permit persons to whom the Software is\r\n * furnished to do so, subject to the following conditions:\r\n *\r\n * The above copyright notice and this permission notice shall be included in\r\n * all copies or substantial portions of the Software.\r\n *\r\n * THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\r\n * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\r\n * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\r\n * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\r\n * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\r\n * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\r\n * SOFTWARE.\r\n */\r\n\r\n/*\r\n * This file adds definitions appropriate for environments in which an unsigned char\r\n * volatile load has acquire semantics, and an unsigned char volatile store has release\r\n * semantics. This is true with the standard Itanium ABI.\r\n */\r\n#if !defined(AO_GCC_BARRIER)\r\n# if defined(__GNUC__)\r\n# define AO_GCC_BARRIER() AO_compiler_barrier()\r\n# else\r\n# define AO_GCC_BARRIER()\r\n# endif\r\n#endif\r\n\r\nAO_INLINE unsigned char\r\nAO_char_load_acquire(const volatile unsigned char *p)\r\n{\r\n unsigned char result = *p;\r\n /* A normal volatile load generates an ld.acq */\r\n AO_GCC_BARRIER();\r\n return result;\r\n}\r\n#define AO_HAVE_char_load_acquire\r\n\r\nAO_INLINE void\r\nAO_char_store_release(volatile unsigned char *p, unsigned char val)\r\n{\r\n AO_GCC_BARRIER();\r\n /* A normal volatile store generates an st.rel */\r\n *p = val;\r\n}\r\n#define AO_HAVE_char_store_release\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/char_atomic_load_store.h", "content": "/*\r\n * Copyright (c) 2003 by Hewlett-Packard Company. All rights reserved.\r\n *\r\n * Permission is hereby granted, free of charge, to any person obtaining a copy\r\n * of this software and associated documentation files (the \"Software\"), to deal\r\n * in the Software without restriction, including without limitation the rights\r\n * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\r\n * copies of the Software, and to permit persons to whom the Software is\r\n * furnished to do so, subject to the following conditions:\r\n *\r\n * The above copyright notice and this permission notice shall be included in\r\n * all copies or substantial portions of the Software.\r\n *\r\n * THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\r\n * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\r\n * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\r\n * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\r\n * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\r\n * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\r\n * SOFTWARE.\r\n */\r\n\r\n/*\r\n * Definitions for architectures on which loads and stores of unsigned char are\r\n * atomic for all legal alignments.\r\n */\r\n\r\nAO_INLINE unsigned char\r\nAO_char_load(const volatile unsigned char *addr)\r\n{\r\n /* Cast away the volatile for architectures like IA64 where */\r\n /* volatile adds barrier semantics. */\r\n return (*(const unsigned char *)addr);\r\n}\r\n\r\n#define AO_HAVE_char_load\r\n\r\nAO_INLINE void\r\nAO_char_store(volatile unsigned char *addr, unsigned char new_val)\r\n{\r\n (*(unsigned char *)addr) = new_val;\r\n}\r\n\r\n#define AO_HAVE_char_store\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/emul_cas.h", "content": "/*\r\n * Copyright (c) 2003 by Hewlett-Packard Company. All rights reserved.\r\n * \r\n * Permission is hereby granted, free of charge, to any person obtaining a copy\r\n * of this software and associated documentation files (the \"Software\"), to deal\r\n * in the Software without restriction, including without limitation the rights\r\n * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\r\n * copies of the Software, and to permit persons to whom the Software is\r\n * furnished to do so, subject to the following conditions:\r\n * \r\n * The above copyright notice and this permission notice shall be included in\r\n * all copies or substantial portions of the Software.\r\n * \r\n * THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\r\n * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\r\n * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\r\n * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\r\n * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\r\n * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\r\n * SOFTWARE. \r\n */\r\n\r\n/*\r\n * Ensure, if at all possible, that AO_compare_and_swap_full() is\r\n * available. The emulation should be brute-force signal-safe, even\r\n * though it actually blocks.\r\n * Including this file will generate an error if AO_compare_and_swap_full()\r\n * cannot be made available.\r\n * This will be included from platform-specific atomic_ops files\r\n * id appropriate, and if AO_FORCE_CAS is defined. It should not be\r\n * included directly, especially since it affects the implementation\r\n * of other atomic update primitives.\r\n * The implementation assumes that only AO_store_XXX and AO_test_and_set_XXX\r\n * variants are defined, and that AO_test_and_set_XXX is not used to\r\n * operate on compare_and_swap locations.\r\n */\r\n\r\n#if !defined(ATOMIC_OPS_H)\r\n# error This file should not be included directly.\r\n#endif\r\n\r\n#ifndef AO_HAVE_double_t\r\n# include \"standard_ao_double_t.h\"\r\n#endif\r\n\r\nint AO_compare_and_swap_emulation(volatile AO_t *addr, AO_t old,\r\n\t\t\t\t AO_t new_val);\r\n\r\nint AO_compare_double_and_swap_double_emulation(volatile AO_double_t *addr,\r\n\t\t\t\t\t\tAO_t old_val1, AO_t old_val2,\r\n\t\t\t\t AO_t new_val1, AO_t new_val2);\r\n\r\nvoid AO_store_full_emulation(volatile AO_t *addr, AO_t val);\r\n\r\n#define AO_compare_and_swap_full(addr, old, newval) \\\r\n\tAO_compare_and_swap_emulation(addr, old, newval)\r\n#define AO_HAVE_compare_and_swap_full\r\n\r\n#ifndef AO_HAVE_compare_double_and_swap_double\r\n# define AO_compare_double_and_swap_double_full(addr, old1, old2, \\\r\n\t\t\t\t\t\tnewval1, newval2) \\\r\n\t AO_compare_double_and_swap_double_emulation(addr, old1, old2, \\\r\n\t\t\t \t\t\t newval1, newval2)\r\n# define AO_HAVE_compare_double_and_swap_double_full\r\n#endif\r\n\r\n#undef AO_store\r\n#undef AO_HAVE_store\r\n#undef AO_store_write\r\n#undef AO_HAVE_store_write\r\n#undef AO_store_release\r\n#undef AO_HAVE_store_release\r\n#undef AO_store_full\r\n#undef AO_HAVE_store_full\r\n#define AO_store_full(addr, val) AO_store_full_emulation(addr, val)\r\n#define AO_HAVE_store_full\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/gcc/alpha.h", "content": "/*\r\n * Copyright (c) 1991-1994 by Xerox Corporation. All rights reserved.\r\n * Copyright (c) 1996-1999 by Silicon Graphics. All rights reserved.\r\n * Copyright (c) 1999-2003 by Hewlett-Packard Company. All rights reserved.\r\n *\r\n *\r\n * THIS MATERIAL IS PROVIDED AS IS, WITH ABSOLUTELY NO WARRANTY EXPRESSED\r\n * OR IMPLIED. ANY USE IS AT YOUR OWN RISK.\r\n *\r\n * Permission is hereby granted to use or copy this program\r\n * for any purpose, provided the above notices are retained on all copies.\r\n * Permission to modify the code and to distribute modified code is granted,\r\n * provided the above notices are retained, and a notice that the code was\r\n * modified is included with the above copyright notice.\r\n *\r\n */\r\n\r\n#include \"../atomic_load_store.h\"\r\n\r\n#include \"../test_and_set_t_is_ao_t.h\"\r\n\r\n#define AO_NO_DD_ORDERING\r\n /* Data dependence does not imply read ordering. */\r\n\r\nAO_INLINE void\r\nAO_nop_full(void)\r\n{\r\n __asm__ __volatile__(\"mb\" : : : \"memory\");\r\n}\r\n\r\n#define AO_HAVE_nop_full\r\n\r\nAO_INLINE void\r\nAO_nop_write(void)\r\n{\r\n __asm__ __volatile__(\"wmb\" : : : \"memory\");\r\n}\r\n\r\n#define AO_HAVE_nop_write\r\n\r\n/* mb should be used for AO_nop_read(). That's the default. */\r\n\r\n/* We believe that ldq_l ... stq_c does not imply any memory barrier. */\r\n/* We should add an explicit fetch_and_add definition. */\r\nAO_INLINE int\r\nAO_compare_and_swap(volatile AO_t *addr,\r\n AO_t old, AO_t new_val)\r\n{\r\n unsigned long was_equal;\r\n unsigned long temp;\r\n\r\n __asm__ __volatile__(\r\n \"1: ldq_l %0,%1\\n\"\r\n \" cmpeq %0,%4,%2\\n\"\r\n \" mov %3,%0\\n\"\r\n \" beq %2,2f\\n\"\r\n \" stq_c %0,%1\\n\"\r\n \" beq %0,1b\\n\"\r\n \"2:\\n\"\r\n :\"=&r\" (temp), \"=m\" (*addr), \"=&r\" (was_equal)\r\n : \"r\" (new_val), \"Ir\" (old)\r\n :\"memory\");\r\n return was_equal;\r\n}\r\n\r\n#define AO_HAVE_compare_and_swap\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/gcc/arm.h", "content": "/*\r\n * Copyright (c) 1991-1994 by Xerox Corporation. All rights reserved.\r\n * Copyright (c) 1996-1999 by Silicon Graphics. All rights reserved.\r\n * Copyright (c) 1999-2003 by Hewlett-Packard Company. All rights reserved.\r\n *\r\n *\r\n * THIS MATERIAL IS PROVIDED AS IS, WITH ABSOLUTELY NO WARRANTY EXPRESSED\r\n * OR IMPLIED. ANY USE IS AT YOUR OWN RISK.\r\n *\r\n * Permission is hereby granted to use or copy this program\r\n * for any purpose, provided the above notices are retained on all copies.\r\n * Permission to modify the code and to distribute modified code is granted,\r\n * provided the above notices are retained, and a notice that the code was\r\n * modified is included with the above copyright notice.\r\n *\r\n */\r\n\r\n#include \"../read_ordered.h\"\r\n\r\n#include \"../test_and_set_t_is_ao_t.h\" /* Probably suboptimal */\r\n\r\n/* NEC LE-IT: ARMv6 is the first architecture providing support for simple LL/SC\r\n * A data memory barrier must be raised via CP15 command (see documentation).\r\n *\r\n * ARMv7 is compatible to ARMv6 but has a simpler command for issuing a\r\n * memory barrier (DMB). Raising it via CP15 should still work as told me by the\r\n * support engineers. If it turns out to be much quicker than we should implement\r\n * custom code for ARMv7 using the asm { dmb } command.\r\n *\r\n * If only a single processor is used, we can define AO_UNIPROCESSOR\r\n * and do not need to access CP15 for ensuring a DMB\r\n*/\r\n\r\n/* NEC LE-IT: gcc has no way to easily check the arm architecture\r\n * but defines only one of __ARM_ARCH_x__ to be true */\r\n#if defined(__ARM_ARCH_6__) || defined(__ARM_ARCH_6J__) \\\r\n || defined(__ARM_ARCH_6K__) || defined(__ARM_ARCH_6ZK__) \\\r\n || defined(__ARM_ARCH_7__) || defined(__ARM_ARCH_7A__) \\\r\n || defined(__ARM_ARCH_7M__) || defined(__ARM_ARCH_7R__)\r\n\r\n#include \"../standard_ao_double_t.h\"\r\n\r\nAO_INLINE void\r\nAO_nop_full(void)\r\n{\r\n#ifndef AO_UNIPROCESSOR\r\n /* issue an data memory barrier (keeps ordering of memory transactions */\r\n /* before and after this operation) */\r\n unsigned int arg=0;\r\n __asm__ __volatile__(\"mcr p15,0,%0,c7,c10,5\" : : \"r\" (arg) : \"memory\");\r\n#endif\r\n}\r\n\r\n#define AO_HAVE_nop_full\r\n\r\n/* NEC LE-IT: AO_t load is simple reading */\r\nAO_INLINE AO_t\r\nAO_load(const volatile AO_t *addr)\r\n{\r\n /* Cast away the volatile for architectures like IA64 where */\r\n /* volatile adds barrier semantics. */\r\n return (*(const AO_t *)addr);\r\n}\r\n#define AO_HAVE_load\r\n\r\n/* NEC LE-IT: atomic \"store\" - according to ARM documentation this is\r\n * the only safe way to set variables also used in LL/SC environment.\r\n * A direct write won't be recognized by the LL/SC construct on the _same_ CPU.\r\n * Support engineers response for behaviour of ARMv6:\r\n *\r\n Core1 Core2 SUCCESS\r\n ===================================\r\n LDREX(x)\r\n STREX(x) Yes\r\n -----------------------------------\r\n LDREX(x)\r\n STR(x)\r\n STREX(x) No\r\n -----------------------------------\r\n LDREX(x)\r\n STR(x)\r\n STREX(x) Yes\r\n -----------------------------------\r\n\r\n * ARMv7 behaves similar, see documentation CortexA8 TRM, point 8.5\r\n *\r\n * HB: I think this is only a problem if interrupt handlers do not clear\r\n * the reservation, as they almost certainly should. Probably change this back\r\n * in a while?\r\n*/\r\nAO_INLINE void AO_store(volatile AO_t *addr, AO_t value)\r\n{\r\n AO_t flag;\r\n\r\n __asm__ __volatile__(\"@AO_store\\n\"\r\n\"1: ldrex %0, [%2]\\n\"\r\n\" strex %0, %3, [%2]\\n\"\r\n\" teq %0, #0\\n\"\r\n\" bne 1b\"\r\n : \"=&r\"(flag), \"+m\"(*addr)\r\n : \"r\" (addr), \"r\"(value)\r\n : \"cc\");\r\n}\r\n#define AO_HAVE_store\r\n\r\n/* NEC LE-IT: replace the SWAP as recommended by ARM:\r\n\r\n \"Applies to: ARM11 Cores\r\n Though the SWP instruction will still work with ARM V6 cores, it is\r\n recommended to use the new V6 synchronization instructions. The SWP\r\n instruction produces 'locked' read and write accesses which are atomic,\r\n i.e. another operation cannot be done between these locked accesses which\r\n ties up external bus (AHB,AXI) bandwidth and can increase worst case\r\n interrupt latencies. LDREX,STREX are more flexible, other instructions can\r\n be done between the LDREX and STREX accesses.\r\n \"\r\n*/\r\nAO_INLINE AO_TS_t\r\nAO_test_and_set(volatile AO_TS_t *addr)\r\n{\r\n\r\n AO_TS_t oldval;\r\n unsigned long flag;\r\n\r\n __asm__ __volatile__(\"@AO_test_and_set\\n\"\r\n\"1: ldrex %0, [%3]\\n\"\r\n\" strex %1, %4, [%3]\\n\"\r\n\" teq %1, #0\\n\"\r\n\" bne 1b\\n\"\r\n : \"=&r\"(oldval),\"=&r\"(flag), \"+m\"(*addr)\r\n : \"r\"(addr), \"r\"(1)\r\n : \"cc\");\r\n\r\n return oldval;\r\n}\r\n\r\n#define AO_HAVE_test_and_set\r\n\r\n/* NEC LE-IT: fetch and add for ARMv6 */\r\nAO_INLINE AO_t\r\nAO_fetch_and_add(volatile AO_t *p, AO_t incr)\r\n{\r\n unsigned long flag,tmp;\r\n AO_t result;\r\n\r\n __asm__ __volatile__(\"@AO_fetch_and_add\\n\"\r\n\"1: ldrex %0, [%5]\\n\" /* get original */\r\n\" add %2, %0, %4\\n\" /* sum up in incr */\r\n\" strex %1, %2, [%5]\\n\" /* store them */\r\n\" teq %1, #0\\n\"\r\n\" bne 1b\\n\"\r\n : \"=&r\"(result),\"=&r\"(flag),\"=&r\"(tmp),\"+m\"(*p) /* 0..3 */\r\n : \"r\"(incr), \"r\"(p) /* 4..5 */\r\n : \"cc\");\r\n\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_fetch_and_add\r\n\r\n/* NEC LE-IT: fetch and add1 for ARMv6 */\r\nAO_INLINE AO_t\r\nAO_fetch_and_add1(volatile AO_t *p)\r\n{\r\n unsigned long flag,tmp;\r\n AO_t result;\r\n\r\n __asm__ __volatile__(\"@AO_fetch_and_add1\\n\"\r\n\"1: ldrex %0, [%4]\\n\" /* get original */\r\n\" add %1, %0, #1\\n\" /* increment */\r\n\" strex %2, %1, [%4]\\n\" /* store them */\r\n\" teq %2, #0\\n\"\r\n\" bne 1b\\n\"\r\n : \"=&r\"(result), \"=&r\"(tmp), \"=&r\"(flag), \"+m\"(*p)\r\n : \"r\"(p)\r\n : \"cc\");\r\n\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_fetch_and_add1\r\n\r\n/* NEC LE-IT: fetch and sub for ARMv6 */\r\nAO_INLINE AO_t\r\nAO_fetch_and_sub1(volatile AO_t *p)\r\n{\r\n unsigned long flag,tmp;\r\n AO_t result;\r\n\r\n __asm__ __volatile__(\"@AO_fetch_and_sub1\\n\"\r\n\"1: ldrex %0, [%4]\\n\" /* get original */\r\n\" sub %1, %0, #1\\n\" /* decrement */\r\n\" strex %2, %1, [%4]\\n\" /* store them */\r\n\" teq %2, #0\\n\"\r\n\" bne 1b\\n\"\r\n : \"=&r\"(result), \"=&r\"(tmp), \"=&r\"(flag), \"+m\"(*p)\r\n : \"r\"(p)\r\n : \"cc\");\r\n\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_fetch_and_sub1\r\n\r\n/* NEC LE-IT: compare and swap */\r\n/* Returns nonzero if the comparison succeeded. */\r\nAO_INLINE int\r\nAO_compare_and_swap(volatile AO_t *addr,\r\n AO_t old_val, AO_t new_val)\r\n{\r\n AO_t result,tmp;\r\n\r\n __asm__ __volatile__(\"@ AO_compare_and_swap\\n\"\r\n\"1: mov %0, #2\\n\" /* store a flag */\r\n\" ldrex %1, [%3]\\n\" /* get original */\r\n\" teq %1, %4\\n\" /* see if match */\r\n\" it eq\\n\"\r\n\" strexeq %0, %5, [%3]\\n\" /* store new one if matched */\r\n\" teq %0, #1\\n\"\r\n\" beq 1b\\n\" /* if update failed, repeat */\r\n : \"=&r\"(result), \"=&r\"(tmp), \"+m\"(*addr)\r\n : \"r\"(addr), \"r\"(old_val), \"r\"(new_val)\r\n : \"cc\");\r\n\r\n return !(result&2); /* if succeded, return 1, else 0 */\r\n}\r\n#define AO_HAVE_compare_and_swap\r\n\r\nAO_INLINE int\r\nAO_compare_double_and_swap_double(volatile AO_double_t *addr,\r\n AO_t old_val1, AO_t old_val2,\r\n AO_t new_val1, AO_t new_val2)\r\n{\r\n double_ptr_storage old_val = ((double_ptr_storage)old_val2 << 32) | old_val1;\r\n double_ptr_storage new_val = ((double_ptr_storage)new_val2 << 32) | new_val1;\r\n\r\n double_ptr_storage tmp;\r\n int result;\r\n\r\n while(1) {\r\n __asm__ __volatile__(\"@ AO_compare_and_swap_double\\n\"\r\n \" ldrexd %0, [%1]\\n\" /* get original to r1 & r2 */\r\n : \"=&r\"(tmp)\r\n : \"r\"(addr)\r\n : \"cc\");\r\n if(tmp != old_val) return 0;\r\n __asm__ __volatile__(\r\n \" strexd %0, %2, [%3]\\n\" /* store new one if matched */\r\n : \"=&r\"(result),\"+m\"(*addr)\r\n : \"r\"(new_val), \"r\"(addr)\r\n : \"cc\");\r\n if(!result) return 1;\r\n }\r\n}\r\n\r\n#define AO_HAVE_compare_double_and_swap_double\r\n\r\n#else\r\n/* pre ARMv6 architectures ... */\r\n\r\n/* I found a slide set that, if I read it correctly, claims that */\r\n/* Loads followed by either a Load or Store are ordered, but nothing */\r\n/* else is. */\r\n/* It appears that SWP is the only simple memory barrier. */\r\n#include \"../all_atomic_load_store.h\"\r\n\r\nAO_INLINE AO_TS_VAL_t\r\nAO_test_and_set_full(volatile AO_TS_t *addr)\r\n{\r\n AO_TS_VAL_t oldval;\r\n /* SWP on ARM is very similar to XCHG on x86. */\r\n /* The first operand is the result, the second the value */\r\n /* to be stored. Both registers must be different from addr. */\r\n /* Make the address operand an early clobber output so it */\r\n /* doesn't overlap with the other operands. The early clobber*/\r\n /* on oldval is necessary to prevent the compiler allocating */\r\n /* them to the same register if they are both unused. */\r\n __asm__ __volatile__(\"swp %0, %2, [%3]\"\r\n : \"=&r\"(oldval), \"=&r\"(addr)\r\n : \"r\"(1), \"1\"(addr)\r\n : \"memory\");\r\n return oldval;\r\n}\r\n\r\n#define AO_HAVE_test_and_set_full\r\n\r\n#endif /* __ARM_ARCH_x */\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/gcc/avr32.h", "content": "/*\r\n * Copyright (C) 2009 Bradley Smith \r\n *\r\n * Permission is hereby granted, free of charge, to any person obtaining a\r\n * copy of this software and associated documentation files (the\r\n * \"Software\"), to deal in the Software without restriction, including\r\n * without limitation the rights to use, copy, modify, merge, publish,\r\n * distribute, sublicense, and/or sell copies of the Software, and to\r\n * permit persons to whom the Software is furnished to do so, subject to\r\n * the following conditions:\r\n *\r\n * The above copyright notice and this permission notice shall be included\r\n * in all copies or substantial portions of the Software.\r\n *\r\n * THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS\r\n * OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF\r\n * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.\r\n * IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY\r\n * CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,\r\n * TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE\r\n * SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\r\n *\r\n */\r\n\r\n#include \"../all_atomic_load_store.h\"\r\n\r\n#include \"../ordered.h\" /* There are no multiprocessor implementations. */\r\n\r\n#include \"../test_and_set_t_is_ao_t.h\"\r\n\r\nAO_INLINE AO_TS_VAL_t\r\nAO_test_and_set_full(volatile AO_TS_t *addr)\r\n{\r\n register long ret;\r\n\r\n __asm__ __volatile__(\r\n \"xchg %[oldval], %[mem], %[newval]\"\r\n : [oldval] \"=&r\"(ret)\r\n : [mem] \"r\"(addr), [newval] \"r\"(1)\r\n : \"memory\");\r\n\r\n return (AO_TS_VAL_t)ret;\r\n}\r\n#define AO_HAVE_test_and_set_full\r\n\r\nAO_INLINE int\r\nAO_compare_and_swap_full(volatile AO_t *addr, AO_t old, AO_t new_val)\r\n{\r\n register long ret;\r\n\r\n __asm__ __volatile__(\r\n \"1: ssrf 5\\n\"\r\n \" ld.w %[res], %[mem]\\n\"\r\n \" eor %[res], %[oldval]\\n\"\r\n \" brne 2f\\n\"\r\n \" stcond %[mem], %[newval]\\n\"\r\n \" brne 1b\\n\"\r\n \"2:\\n\"\r\n : [res] \"=&r\"(ret), [mem] \"=m\"(*addr)\r\n : \"m\"(*addr), [newval] \"r\"(new_val), [oldval] \"r\"(old)\r\n : \"cc\", \"memory\");\r\n\r\n return (int)ret;\r\n}\r\n#define AO_HAVE_compare_and_swap_full\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/gcc/cris.h", "content": "/*\r\n * Copyright (c) 2004 Hewlett-Packard Development Company, L.P.\r\n * \r\n * Permission is hereby granted, free of charge, to any person obtaining a copy\r\n * of this software and associated documentation files (the \"Software\"), to deal\r\n * in the Software without restriction, including without limitation the rights\r\n * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\r\n * copies of the Software, and to permit persons to whom the Software is\r\n * furnished to do so, subject to the following conditions:\r\n * \r\n * The above copyright notice and this permission notice shall be included in\r\n * all copies or substantial portions of the Software.\r\n * \r\n * THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\r\n * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\r\n * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\r\n * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\r\n * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\r\n * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\r\n * SOFTWARE. \r\n *\r\n * Most of this code originally comes from Hans-Peter Nilsson. It is included\r\n * here with his permission.\r\n *\r\n * This version has not been tested. It was coped here from a GC\r\n * patch so that we wouldn't lose the code in the upgrade to gc7.\r\n */ \r\n\r\n#include \"../all_atomic_load_store.h\"\r\n\r\n#include \"../ordered.h\" /* There are no multiprocessor implementations. */\r\n\r\n#include \"../test_and_set_t_is_ao_t.h\"\r\n\r\n/*\r\n * The architecture apparently supports an \"f\" flag which is\r\n * set on preemption. This essentially gives us load-locked,\r\n * store-conditional primitives, though I'm not quite sure how\r\n * this would work on a hypothetical multiprocessor. -HB\r\n *\r\n * For details, see\r\n * http://developer.axis.com/doc/hardware/etrax100lx/prog_man/\r\n * 1_architectural_description.pdf\r\n *\r\n * Presumably many other primitives (notably CAS, including the double-\r\n * width versions) could be implemented in this manner, if someone got\r\n * around to it.\r\n */\r\n\r\nAO_INLINE AO_TS_VAL_t\r\nAO_test_and_set_full(volatile AO_TS_t *addr) {\r\n /* Ripped from linuxthreads/sysdeps/cris/pt-machine.h */\r\n register unsigned long int ret;\r\n\r\n /* Note the use of a dummy output of *addr to expose the write. The\r\n memory barrier is to stop *other* writes being moved past this code. */\r\n __asm__ __volatile__(\"clearf\\n\"\r\n \t\t \"0:\\n\\t\"\r\n \t \"movu.b [%2],%0\\n\\t\"\r\n \t \"ax\\n\\t\"\r\n \t \"move.b %3,[%2]\\n\\t\"\r\n \t \"bwf 0b\\n\\t\"\r\n \t \"clearf\"\r\n \t : \"=&r\" (ret), \"=m\" (*addr)\r\n \t : \"r\" (addr), \"r\" ((int) 1), \"m\" (*addr)\r\n \t : \"memory\");\r\n return ret;\r\n}\r\n\r\n#define AO_HAVE_test_and_set_full\r\n\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/gcc/hppa.h", "content": "/*\r\n * Copyright (c) 2003 Hewlett-Packard Development Company, L.P.\r\n *\r\n * Permission is hereby granted, free of charge, to any person obtaining a copy\r\n * of this software and associated documentation files (the \"Software\"), to deal\r\n * in the Software without restriction, including without limitation the rights\r\n * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\r\n * copies of the Software, and to permit persons to whom the Software is\r\n * furnished to do so, subject to the following conditions:\r\n *\r\n * The above copyright notice and this permission notice shall be included in\r\n * all copies or substantial portions of the Software.\r\n *\r\n * THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\r\n * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\r\n * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\r\n * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\r\n * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\r\n * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\r\n * SOFTWARE.\r\n *\r\n * Modified by Carlos O'Donell , 2003\r\n * - Added self-aligning lock.\r\n *\r\n */\r\n\r\n#include \"../all_atomic_load_store.h\"\r\n\r\n/* Some architecture set descriptions include special \"ordered\" memory */\r\n/* operations. As far as we can tell, no existing processors actually */\r\n/* require those. Nor does it appear likely that future processors */\r\n/* will. */\r\n#include \"../ordered.h\"\r\n\r\n/* GCC will not guarantee the alignment we need, use four lock words */\r\n/* and select the correctly aligned datum. See the glibc 2.3.2 */\r\n/* linuxthread port for the original implementation. */\r\nstruct AO_pa_clearable_loc {\r\n int data[4];\r\n};\r\n\r\n#undef AO_TS_INITIALIZER\r\n#define AO_TS_t struct AO_pa_clearable_loc\r\n#define AO_TS_INITIALIZER {1,1,1,1}\r\n/* Switch meaning of set and clear, since we only have an atomic clear */\r\n/* instruction. */\r\ntypedef enum {AO_PA_TS_set = 0, AO_PA_TS_clear = 1} AO_PA_TS_val;\r\n#define AO_TS_VAL_t AO_PA_TS_val\r\n#define AO_TS_CLEAR AO_PA_TS_clear\r\n#define AO_TS_SET AO_PA_TS_set\r\n\r\n/* The hppa only has one atomic read and modify memory operation, */\r\n/* load and clear, so hppa spinlocks must use zero to signify that */\r\n/* someone is holding the lock. The address used for the ldcw */\r\n/* semaphore must be 16-byte aligned. */\r\n\r\n#define __ldcw(a) ({ \\\r\n volatile unsigned int __ret; \\\r\n __asm__ __volatile__(\"ldcw 0(%2),%0\" \\\r\n : \"=r\" (__ret), \"=m\" (*(a)) : \"r\" (a)); \\\r\n __ret; \\\r\n})\r\n\r\n/* Because malloc only guarantees 8-byte alignment for malloc'd data, */\r\n/* and GCC only guarantees 8-byte alignment for stack locals, we can't */\r\n/* be assured of 16-byte alignment for atomic lock data even if we */\r\n/* specify \"__attribute ((aligned(16)))\" in the type declaration. So, */\r\n/* we use a struct containing an array of four ints for the atomic lock */\r\n/* type and dynamically select the 16-byte aligned int from the array */\r\n/* for the semaphore. */\r\n#define __PA_LDCW_ALIGNMENT 16\r\n#define __ldcw_align(a) ({ \\\r\n unsigned long __ret = (unsigned long) a; \\\r\n __ret += __PA_LDCW_ALIGNMENT - 1; \\\r\n __ret &= ~(__PA_LDCW_ALIGNMENT - 1); \\\r\n (volatile unsigned int *) __ret; \\\r\n})\r\n\r\n/* Works on PA 1.1 and PA 2.0 systems */\r\nAO_INLINE AO_TS_VAL_t\r\nAO_test_and_set_full(volatile AO_TS_t * addr)\r\n{\r\n volatile unsigned int *a = __ldcw_align (addr);\r\n return (AO_TS_VAL_t) __ldcw (a);\r\n}\r\n\r\nAO_INLINE void\r\nAO_pa_clear(volatile AO_TS_t * addr)\r\n{\r\n volatile unsigned int *a = __ldcw_align (addr);\r\n AO_compiler_barrier();\r\n *a = 1;\r\n}\r\n#define AO_CLEAR(addr) AO_pa_clear(addr)\r\n\r\n#define AO_HAVE_test_and_set_full\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/gcc/ia64.h", "content": "/*\r\n * Copyright (c) 2003 Hewlett-Packard Development Company, L.P.\r\n *\r\n * Permission is hereby granted, free of charge, to any person obtaining a copy\r\n * of this software and associated documentation files (the \"Software\"), to deal\r\n * in the Software without restriction, including without limitation the rights\r\n * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\r\n * copies of the Software, and to permit persons to whom the Software is\r\n * furnished to do so, subject to the following conditions:\r\n *\r\n * The above copyright notice and this permission notice shall be included in\r\n * all copies or substantial portions of the Software.\r\n *\r\n * THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\r\n * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\r\n * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\r\n * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\r\n * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\r\n * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\r\n * SOFTWARE.\r\n */\r\n\r\n#include \"../all_atomic_load_store.h\"\r\n\r\n#include \"../all_acquire_release_volatile.h\"\r\n\r\n#include \"../test_and_set_t_is_char.h\"\r\n\r\n#ifdef _ILP32\r\n /* 32-bit HP/UX code. */\r\n /* This requires pointer \"swizzling\". Pointers need to be expanded */\r\n /* to 64 bits using the addp4 instruction before use. This makes it */\r\n /* hard to share code, but we try anyway. */\r\n# define AO_LEN \"4\"\r\n /* We assume that addr always appears in argument position 1 in asm */\r\n /* code. If it is clobbered due to swizzling, we also need it in */\r\n /* second position. Any later arguments are referenced symbolically, */\r\n /* so that we don't have to worry about their position. This requires*/\r\n /* gcc 3.1, but you shouldn't be using anything older than that on */\r\n /* IA64 anyway. */\r\n /* The AO_MASK macro is a workaround for the fact that HP/UX gcc */\r\n /* appears to otherwise store 64-bit pointers in ar.ccv, i.e. it */\r\n /* doesn't appear to clear high bits in a pointer value we pass into */\r\n /* assembly code, even if it is supposedly of type AO_t. */\r\n# define AO_IN_ADDR \"1\"(addr)\r\n# define AO_OUT_ADDR , \"=r\"(addr)\r\n# define AO_SWIZZLE \"addp4 %1=0,%1;;\\n\"\r\n# define AO_MASK(ptr) __asm__(\"zxt4 %1=%1\": \"=r\"(ptr) : \"0\"(ptr));\r\n#else\r\n# define AO_LEN \"8\"\r\n# define AO_IN_ADDR \"r\"(addr)\r\n# define AO_OUT_ADDR\r\n# define AO_SWIZZLE\r\n# define AO_MASK(ptr)\r\n#endif\r\n\r\nAO_INLINE void\r\nAO_nop_full(void)\r\n{\r\n __asm__ __volatile__(\"mf\" : : : \"memory\");\r\n}\r\n#define AO_HAVE_nop_full\r\n\r\nAO_INLINE AO_t\r\nAO_fetch_and_add1_acquire (volatile AO_t *addr)\r\n{\r\n AO_t result;\r\n\r\n __asm__ __volatile__ (AO_SWIZZLE\r\n \"fetchadd\" AO_LEN \".acq %0=[%1],1\":\r\n \"=r\" (result) AO_OUT_ADDR: AO_IN_ADDR :\"memory\");\r\n return result;\r\n}\r\n#define AO_HAVE_fetch_and_add1_acquire\r\n\r\nAO_INLINE AO_t\r\nAO_fetch_and_add1_release (volatile AO_t *addr)\r\n{\r\n AO_t result;\r\n\r\n __asm__ __volatile__ (AO_SWIZZLE\r\n \"fetchadd\" AO_LEN \".rel %0=[%1],1\":\r\n \"=r\" (result) AO_OUT_ADDR: AO_IN_ADDR :\"memory\");\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_fetch_and_add1_release\r\n\r\nAO_INLINE AO_t\r\nAO_fetch_and_sub1_acquire (volatile AO_t *addr)\r\n{\r\n AO_t result;\r\n\r\n __asm__ __volatile__ (AO_SWIZZLE\r\n \"fetchadd\" AO_LEN \".acq %0=[%1],-1\":\r\n \"=r\" (result) AO_OUT_ADDR: AO_IN_ADDR :\"memory\");\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_fetch_and_sub1_acquire\r\n\r\nAO_INLINE AO_t\r\nAO_fetch_and_sub1_release (volatile AO_t *addr)\r\n{\r\n AO_t result;\r\n\r\n __asm__ __volatile__ (AO_SWIZZLE\r\n \"fetchadd\" AO_LEN \".rel %0=[%1],-1\":\r\n \"=r\" (result) AO_OUT_ADDR: AO_IN_ADDR :\"memory\");\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_fetch_and_sub1_release\r\n\r\n#ifndef _ILP32\r\n\r\nAO_INLINE unsigned int\r\nAO_int_fetch_and_add1_acquire (volatile unsigned int *addr)\r\n{\r\n unsigned int result;\r\n\r\n __asm__ __volatile__ (\"fetchadd4.acq %0=[%1],1\":\r\n \"=r\" (result): AO_IN_ADDR :\"memory\");\r\n return result;\r\n}\r\n#define AO_HAVE_int_fetch_and_add1_acquire\r\n\r\nAO_INLINE unsigned int\r\nAO_int_fetch_and_add1_release (volatile unsigned int *addr)\r\n{\r\n unsigned int result;\r\n\r\n __asm__ __volatile__ (\"fetchadd4.rel %0=[%1],1\":\r\n \"=r\" (result): AO_IN_ADDR :\"memory\");\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_int_fetch_and_add1_release\r\n\r\nAO_INLINE unsigned int\r\nAO_int_fetch_and_sub1_acquire (volatile unsigned int *addr)\r\n{\r\n unsigned int result;\r\n\r\n __asm__ __volatile__ (\"fetchadd4.acq %0=[%1],-1\":\r\n \"=r\" (result): AO_IN_ADDR :\"memory\");\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_int_fetch_and_sub1_acquire\r\n\r\nAO_INLINE unsigned int\r\nAO_int_fetch_and_sub1_release (volatile unsigned int *addr)\r\n{\r\n unsigned int result;\r\n\r\n __asm__ __volatile__ (\"fetchadd4.rel %0=[%1],-1\":\r\n \"=r\" (result): AO_IN_ADDR :\"memory\");\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_int_fetch_and_sub1_release\r\n\r\n#endif /* !_ILP32 */\r\n\r\nAO_INLINE int\r\nAO_compare_and_swap_acquire(volatile AO_t *addr,\r\n AO_t old, AO_t new_val)\r\n{\r\n AO_t oldval;\r\n AO_MASK(old);\r\n __asm__ __volatile__(AO_SWIZZLE\r\n \"mov ar.ccv=%[old] ;; cmpxchg\" AO_LEN\r\n \".acq %0=[%1],%[new_val],ar.ccv\"\r\n : \"=r\"(oldval) AO_OUT_ADDR\r\n : AO_IN_ADDR, [new_val]\"r\"(new_val), [old]\"r\"(old)\r\n : \"memory\");\r\n return (oldval == old);\r\n}\r\n\r\n#define AO_HAVE_compare_and_swap_acquire\r\n\r\nAO_INLINE int\r\nAO_compare_and_swap_release(volatile AO_t *addr,\r\n AO_t old, AO_t new_val)\r\n{\r\n AO_t oldval;\r\n AO_MASK(old);\r\n __asm__ __volatile__(AO_SWIZZLE\r\n \"mov ar.ccv=%[old] ;; cmpxchg\" AO_LEN\r\n \".rel %0=[%1],%[new_val],ar.ccv\"\r\n : \"=r\"(oldval) AO_OUT_ADDR\r\n : AO_IN_ADDR, [new_val]\"r\"(new_val), [old]\"r\"(old)\r\n : \"memory\");\r\n return (oldval == old);\r\n}\r\n\r\n#define AO_HAVE_compare_and_swap_release\r\n\r\nAO_INLINE int\r\nAO_char_compare_and_swap_acquire(volatile unsigned char *addr,\r\n unsigned char old, unsigned char new_val)\r\n{\r\n unsigned char oldval;\r\n __asm__ __volatile__(AO_SWIZZLE\r\n \"mov ar.ccv=%[old] ;; cmpxchg1.acq %0=[%1],%[new_val],ar.ccv\"\r\n : \"=r\"(oldval) AO_OUT_ADDR\r\n : AO_IN_ADDR, [new_val]\"r\"(new_val), [old]\"r\"((AO_t)old)\r\n : \"memory\");\r\n return (oldval == old);\r\n}\r\n\r\n#define AO_HAVE_char_compare_and_swap_acquire\r\n\r\nAO_INLINE int\r\nAO_char_compare_and_swap_release(volatile unsigned char *addr,\r\n unsigned char old, unsigned char new_val)\r\n{\r\n unsigned char oldval;\r\n __asm__ __volatile__(AO_SWIZZLE\r\n \"mov ar.ccv=%[old] ;; cmpxchg1.rel %0=[%1],%[new_val],ar.ccv\"\r\n : \"=r\"(oldval) AO_OUT_ADDR\r\n : AO_IN_ADDR, [new_val]\"r\"(new_val), [old]\"r\"((AO_t)old)\r\n : \"memory\");\r\n return (oldval == old);\r\n}\r\n\r\n#define AO_HAVE_char_compare_and_swap_release\r\n\r\nAO_INLINE int\r\nAO_short_compare_and_swap_acquire(volatile unsigned short *addr,\r\n unsigned short old, unsigned short new_val)\r\n{\r\n unsigned short oldval;\r\n __asm__ __volatile__(AO_SWIZZLE\r\n \"mov ar.ccv=%[old] ;; cmpxchg2.acq %0=[%1],%[new_val],ar.ccv\"\r\n : \"=r\"(oldval) AO_OUT_ADDR\r\n : AO_IN_ADDR, [new_val]\"r\"(new_val), [old]\"r\"((AO_t)old)\r\n : \"memory\");\r\n return (oldval == old);\r\n}\r\n\r\n#define AO_HAVE_short_compare_and_swap_acquire\r\n\r\nAO_INLINE int\r\nAO_short_compare_and_swap_release(volatile unsigned short *addr,\r\n unsigned short old, unsigned short new_val)\r\n{\r\n unsigned short oldval;\r\n __asm__ __volatile__(AO_SWIZZLE\r\n \"mov ar.ccv=%[old] ;; cmpxchg2.rel %0=[%1],%[new_val],ar.ccv\"\r\n : \"=r\"(oldval) AO_OUT_ADDR\r\n : AO_IN_ADDR, [new_val]\"r\"(new_val), [old]\"r\"((AO_t)old)\r\n : \"memory\");\r\n return (oldval == old);\r\n}\r\n\r\n#define AO_HAVE_short_compare_and_swap_release\r\n\r\n#ifndef _ILP32\r\n\r\nAO_INLINE int\r\nAO_int_compare_and_swap_acquire(volatile unsigned int *addr,\r\n unsigned int old, unsigned int new_val)\r\n{\r\n unsigned int oldval;\r\n __asm__ __volatile__(\"mov ar.ccv=%3 ;; cmpxchg4.acq %0=[%1],%2,ar.ccv\"\r\n : \"=r\"(oldval)\r\n : AO_IN_ADDR, \"r\"(new_val), \"r\"((AO_t)old) : \"memory\");\r\n return (oldval == old);\r\n}\r\n\r\n#define AO_HAVE_int_compare_and_swap_acquire\r\n\r\nAO_INLINE int\r\nAO_int_compare_and_swap_release(volatile unsigned int *addr,\r\n unsigned int old, unsigned int new_val)\r\n{\r\n unsigned int oldval;\r\n __asm__ __volatile__(\"mov ar.ccv=%3 ;; cmpxchg4.rel %0=[%1],%2,ar.ccv\"\r\n : \"=r\"(oldval)\r\n : AO_IN_ADDR, \"r\"(new_val), \"r\"((AO_t)old) : \"memory\");\r\n return (oldval == old);\r\n}\r\n\r\n#define AO_HAVE_int_compare_and_swap_release\r\n\r\n#endif /* !_ILP32 */\r\n\r\n/* FIXME: Add compare_and_swap_double as soon as there is widely */\r\n/* available hardware that implements it. */\r\n\r\n/* FIXME: Add compare_double_and_swap_double for the _ILP32 case. */\r\n\r\n#ifdef _ILP32\r\n# include \"../ao_t_is_int.h\"\r\n#endif\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/gcc/m68k.h", "content": "/*\r\n * Copyright (c) 1991-1994 by Xerox Corporation. All rights reserved.\r\n * Copyright (c) 1996-1999 by Silicon Graphics. All rights reserved.\r\n * Copyright (c) 1999-2003 by Hewlett-Packard Company. All rights reserved.\r\n *\r\n *\r\n * THIS MATERIAL IS PROVIDED AS IS, WITH ABSOLUTELY NO WARRANTY EXPRESSED\r\n * OR IMPLIED. ANY USE IS AT YOUR OWN RISK.\r\n *\r\n * Permission is hereby granted to use or copy this program\r\n * for any purpose, provided the above notices are retained on all copies.\r\n * Permission to modify the code and to distribute modified code is granted,\r\n * provided the above notices are retained, and a notice that the code was\r\n * modified is included with the above copyright notice.\r\n *\r\n */\r\n\r\n/* The cas instruction causes an emulation trap for the */\r\n/* 060 with a misaligned pointer, so let's avoid this. */\r\n#undef AO_t\r\ntypedef unsigned long AO_t __attribute__ ((aligned (4)));\r\n\r\n/* FIXME. Very incomplete. */\r\n#include \"../all_aligned_atomic_load_store.h\"\r\n\r\n/* Are there any m68k multiprocessors still around? */\r\n/* AFAIK, Alliants were sequentially consistent. */\r\n#include \"../ordered.h\"\r\n\r\n#include \"../test_and_set_t_is_char.h\"\r\n\r\n/* Contributed by Tony Mantler or new. Should be changed to MIT license? */\r\nAO_INLINE AO_TS_VAL_t\r\nAO_test_and_set_full(volatile AO_TS_t *addr) {\r\n AO_TS_t oldval;\r\n\r\n /* The value at addr is semi-phony. */\r\n /* 'tas' sets bit 7 while the return */\r\n /* value pretends all bits were set, */\r\n /* which at least matches AO_TS_SET. */\r\n __asm__ __volatile__(\r\n \"tas %1; sne %0\"\r\n : \"=d\" (oldval), \"=m\" (*addr)\r\n : \"m\" (*addr)\r\n : \"memory\");\r\n return oldval;\r\n}\r\n\r\n#define AO_HAVE_test_and_set_full\r\n\r\n/* Returns nonzero if the comparison succeeded. */\r\nAO_INLINE int\r\nAO_compare_and_swap_full(volatile AO_t *addr,\r\n AO_t old, AO_t new_val)\r\n{\r\n char result;\r\n\r\n __asm__ __volatile__(\r\n \"cas.l %3,%4,%1; seq %0\"\r\n : \"=d\" (result), \"=m\" (*addr)\r\n : \"m\" (*addr), \"d\" (old), \"d\" (new_val)\r\n : \"memory\");\r\n return -result;\r\n}\r\n\r\n#define AO_HAVE_compare_and_swap_full\r\n\r\n\r\n#include \"../ao_t_is_int.h\"\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/gcc/mips.h", "content": "/*\r\n * Copyright (c) 2005,2007 Thiemo Seufer \r\n *\r\n * THIS MATERIAL IS PROVIDED AS IS, WITH ABSOLUTELY NO WARRANTY EXPRESSED\r\n * OR IMPLIED. ANY USE IS AT YOUR OWN RISK.\r\n *\r\n * Permission is hereby granted to use or copy this program\r\n * for any purpose, provided the above notices are retained on all copies.\r\n * Permission to modify the code and to distribute modified code is granted,\r\n * provided the above notices are retained, and a notice that the code was\r\n * modified is included with the above copyright notice.\r\n */\r\n\r\n/*\r\n * FIXME: This should probably make finer distinctions. SGI MIPS is\r\n * much more strongly ordered, and in fact closer to sequentially\r\n * consistent. This is really aimed at modern embedded implementations.\r\n * It looks to me like this assumes a 32-bit ABI. -HB\r\n */\r\n\r\n#include \"../all_aligned_atomic_load_store.h\"\r\n#include \"../acquire_release_volatile.h\"\r\n#include \"../test_and_set_t_is_ao_t.h\"\r\n#include \"../standard_ao_double_t.h\"\r\n\r\n/* Data dependence does not imply read ordering. */\r\n#define AO_NO_DD_ORDERING\r\n\r\nAO_INLINE void\r\nAO_nop_full(void)\r\n{\r\n __asm__ __volatile__(\r\n \" .set push \\n\"\r\n \" .set mips2 \\n\"\r\n \" .set noreorder \\n\"\r\n \" .set nomacro \\n\"\r\n \" sync \\n\"\r\n \" .set pop \"\r\n : : : \"memory\");\r\n}\r\n\r\n#define AO_HAVE_nop_full\r\n\r\nAO_INLINE int\r\nAO_compare_and_swap(volatile AO_t *addr, AO_t old, AO_t new_val)\r\n{\r\n register int was_equal = 0;\r\n register int temp;\r\n\r\n __asm__ __volatile__(\r\n \" .set push \\n\"\r\n \" .set mips2 \\n\"\r\n \" .set noreorder \\n\"\r\n \" .set nomacro \\n\"\r\n \"1: ll %0, %1 \\n\"\r\n \" bne %0, %4, 2f \\n\"\r\n \" move %0, %3 \\n\"\r\n \" sc %0, %1 \\n\"\r\n \" .set pop \\n\"\r\n \" beqz %0, 1b \\n\"\r\n \" li %2, 1 \\n\"\r\n \"2: \"\r\n : \"=&r\" (temp), \"+R\" (*addr), \"+r\" (was_equal)\r\n : \"r\" (new_val), \"r\" (old)\r\n : \"memory\");\r\n return was_equal;\r\n}\r\n\r\n#define AO_HAVE_compare_and_swap\r\n\r\n/* FIXME: I think the implementations below should be automatically */\r\n/* generated if we omit them. - HB */\r\n\r\nAO_INLINE int\r\nAO_compare_and_swap_acquire(volatile AO_t *addr, AO_t old, AO_t new_val) {\r\n int result = AO_compare_and_swap(addr, old, new_val);\r\n AO_nop_full();\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_compare_and_swap_acquire\r\n\r\nAO_INLINE int\r\nAO_compare_and_swap_release(volatile AO_t *addr, AO_t old, AO_t new_val) {\r\n AO_nop_full();\r\n return AO_compare_and_swap(addr, old, new_val);\r\n}\r\n\r\n#define AO_HAVE_compare_and_swap_release\r\n\r\nAO_INLINE int\r\nAO_compare_and_swap_full(volatile AO_t *addr, AO_t old, AO_t new_val) {\r\n AO_t result;\r\n AO_nop_full();\r\n result = AO_compare_and_swap(addr, old, new_val);\r\n AO_nop_full();\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_compare_and_swap_full\r\n\r\n/*\r\n * FIXME: We should also implement fetch_and_add and or primitives\r\n * directly.\r\n */\r\n\r\n#include \"../ao_t_is_int.h\"\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/gcc/powerpc.h", "content": "/*\r\n * Copyright (c) 1991-1994 by Xerox Corporation. All rights reserved.\r\n * Copyright (c) 1996-1999 by Silicon Graphics. All rights reserved.\r\n * Copyright (c) 1999-2004 Hewlett-Packard Development Company, L.P.\r\n *\r\n *\r\n * THIS MATERIAL IS PROVIDED AS IS, WITH ABSOLUTELY NO WARRANTY EXPRESSED\r\n * OR IMPLIED. ANY USE IS AT YOUR OWN RISK.\r\n *\r\n * Permission is hereby granted to use or copy this program\r\n * for any purpose, provided the above notices are retained on all copies.\r\n * Permission to modify the code and to distribute modified code is granted,\r\n * provided the above notices are retained, and a notice that the code was\r\n * modified is included with the above copyright notice.\r\n *\r\n */\r\n\r\n/* Memory model documented at http://www-106.ibm.com/developerworks/ */\r\n/* eserver/articles/archguide.html and (clearer) */\r\n/* http://www-106.ibm.com/developerworks/eserver/articles/powerpc.html. */\r\n/* There appears to be no implicit ordering between any kind of */\r\n/* independent memory references. */\r\n/* Architecture enforces some ordering based on control dependence. */\r\n/* I don't know if that could help. */\r\n/* Data-dependent loads are always ordered. */\r\n/* Based on the above references, eieio is intended for use on */\r\n/* uncached memory, which we don't support. It does not order loads */\r\n/* from cached memory. */\r\n/* Thanks to Maged Michael, Doug Lea, and Roger Hoover for helping to */\r\n/* track some of this down and correcting my misunderstandings. -HB */\r\n/* Earl Chew subsequently contributed further fixes & additions. */\r\n\r\n#include \"../all_aligned_atomic_load_store.h\"\r\n\r\n#include \"../test_and_set_t_is_ao_t.h\"\r\n /* There seems to be no byte equivalent of lwarx, so this */\r\n /* may really be what we want, at least in the 32-bit case. */\r\n\r\nAO_INLINE void\r\nAO_nop_full(void)\r\n{\r\n __asm__ __volatile__(\"sync\" : : : \"memory\");\r\n}\r\n\r\n#define AO_HAVE_nop_full\r\n\r\n/* lwsync apparently works for everything but a StoreLoad barrier. */\r\nAO_INLINE void\r\nAO_lwsync(void)\r\n{\r\n#ifdef __NO_LWSYNC__\r\n __asm__ __volatile__(\"sync\" : : : \"memory\");\r\n#else\r\n __asm__ __volatile__(\"lwsync\" : : : \"memory\");\r\n#endif\r\n}\r\n\r\n#define AO_nop_write() AO_lwsync()\r\n#define AO_HAVE_nop_write\r\n\r\n#define AO_nop_read() AO_lwsync()\r\n#define AO_HAVE_nop_read\r\n\r\n/* We explicitly specify load_acquire, since it is important, and can */\r\n/* be implemented relatively cheaply. It could be implemented */\r\n/* with an ordinary load followed by a lwsync. But the general wisdom */\r\n/* seems to be that a data dependent branch followed by an isync is */\r\n/* cheaper. And the documentation is fairly explicit that this also */\r\n/* has acquire semantics. */\r\n/* ppc64 uses ld not lwz */\r\n#if defined(__powerpc64__) || defined(__ppc64__) || defined(__64BIT__)\r\nAO_INLINE AO_t\r\nAO_load_acquire(const volatile AO_t *addr)\r\n{\r\n AO_t result;\r\n\r\n __asm__ __volatile__ (\r\n \"ld%U1%X1 %0,%1\\n\"\r\n \"cmpw %0,%0\\n\"\r\n \"bne- 1f\\n\"\r\n \"1: isync\\n\"\r\n : \"=r\" (result)\r\n : \"m\"(*addr) : \"memory\", \"cr0\");\r\n return result;\r\n}\r\n#else\r\nAO_INLINE AO_t\r\nAO_load_acquire(const volatile AO_t *addr)\r\n{\r\n AO_t result;\r\n\r\n /* FIXME: We should get gcc to allocate one of the condition */\r\n /* registers. I always got \"impossible constraint\" when I */\r\n /* tried the \"y\" constraint. */\r\n __asm__ __volatile__ (\r\n \"lwz%U1%X1 %0,%1\\n\"\r\n \"cmpw %0,%0\\n\"\r\n \"bne- 1f\\n\"\r\n \"1: isync\\n\"\r\n : \"=r\" (result)\r\n : \"m\"(*addr) : \"memory\", \"cc\");\r\n return result;\r\n}\r\n#endif\r\n#define AO_HAVE_load_acquire\r\n\r\n/* We explicitly specify store_release, since it relies */\r\n/* on the fact that lwsync is also a LoadStore barrier. */\r\nAO_INLINE void\r\nAO_store_release(volatile AO_t *addr, AO_t value)\r\n{\r\n AO_lwsync();\r\n *addr = value;\r\n}\r\n\r\n#define AO_HAVE_load_acquire\r\n\r\n/* This is similar to the code in the garbage collector. Deleting */\r\n/* this and having it synthesized from compare_and_swap would probably */\r\n/* only cost us a load immediate instruction. */\r\n#if defined(__powerpc64__) || defined(__ppc64__) || defined(__64BIT__)\r\n/* Completely untested. And we should be using smaller objects anyway. */\r\nAO_INLINE AO_TS_VAL_t\r\nAO_test_and_set(volatile AO_TS_t *addr) {\r\n unsigned long oldval;\r\n unsigned long temp = 1; /* locked value */\r\n\r\n __asm__ __volatile__(\r\n \"1:ldarx %0,0,%1\\n\" /* load and reserve */\r\n \"cmpdi %0, 0\\n\" /* if load is */\r\n \"bne 2f\\n\" /* non-zero, return already set */\r\n \"stdcx. %2,0,%1\\n\" /* else store conditional */\r\n \"bne- 1b\\n\" /* retry if lost reservation */\r\n \"2:\\n\" /* oldval is zero if we set */\r\n : \"=&r\"(oldval)\r\n : \"r\"(addr), \"r\"(temp)\r\n : \"memory\", \"cr0\");\r\n\r\n return (AO_TS_VAL_t)oldval;\r\n}\r\n\r\n#else\r\n\r\nAO_INLINE AO_TS_VAL_t\r\nAO_test_and_set(volatile AO_TS_t *addr) {\r\n int oldval;\r\n int temp = 1; /* locked value */\r\n\r\n __asm__ __volatile__(\r\n \"1:lwarx %0,0,%1\\n\" /* load and reserve */\r\n \"cmpwi %0, 0\\n\" /* if load is */\r\n \"bne 2f\\n\" /* non-zero, return already set */\r\n \"stwcx. %2,0,%1\\n\" /* else store conditional */\r\n \"bne- 1b\\n\" /* retry if lost reservation */\r\n \"2:\\n\" /* oldval is zero if we set */\r\n : \"=&r\"(oldval)\r\n : \"r\"(addr), \"r\"(temp)\r\n : \"memory\", \"cr0\");\r\n\r\n return (AO_TS_VAL_t)oldval;\r\n}\r\n\r\n#endif\r\n\r\n#define AO_HAVE_test_and_set\r\n\r\nAO_INLINE AO_TS_VAL_t\r\nAO_test_and_set_acquire(volatile AO_TS_t *addr) {\r\n AO_TS_VAL_t result = AO_test_and_set(addr);\r\n AO_lwsync();\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_test_and_set_acquire\r\n\r\nAO_INLINE AO_TS_VAL_t\r\nAO_test_and_set_release(volatile AO_TS_t *addr) {\r\n AO_lwsync();\r\n return AO_test_and_set(addr);\r\n}\r\n\r\n#define AO_HAVE_test_and_set_release\r\n\r\nAO_INLINE AO_TS_VAL_t\r\nAO_test_and_set_full(volatile AO_TS_t *addr) {\r\n AO_TS_VAL_t result;\r\n AO_lwsync();\r\n result = AO_test_and_set(addr);\r\n AO_lwsync();\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_test_and_set_full\r\n\r\n#if defined(__powerpc64__) || defined(__ppc64__) || defined(__64BIT__)\r\n/* FIXME: Completely untested. */\r\nAO_INLINE int\r\nAO_compare_and_swap(volatile AO_t *addr, AO_t old, AO_t new_val) {\r\n AO_t oldval;\r\n int result = 0;\r\n\r\n __asm__ __volatile__(\r\n \"1:ldarx %0,0,%2\\n\" /* load and reserve */\r\n \"cmpd %0, %4\\n\" /* if load is not equal to */\r\n \"bne 2f\\n\" /* old, fail */\r\n \"stdcx. %3,0,%2\\n\" /* else store conditional */\r\n \"bne- 1b\\n\" /* retry if lost reservation */\r\n \"li %1,1\\n\" /* result = 1; */\r\n \"2:\\n\"\r\n : \"=&r\"(oldval), \"=&r\"(result)\r\n : \"r\"(addr), \"r\"(new_val), \"r\"(old), \"1\"(result)\r\n : \"memory\", \"cr0\");\r\n\r\n return result;\r\n}\r\n\r\n#else\r\n\r\nAO_INLINE int\r\nAO_compare_and_swap(volatile AO_t *addr, AO_t old, AO_t new_val) {\r\n AO_t oldval;\r\n int result = 0;\r\n\r\n __asm__ __volatile__(\r\n \"1:lwarx %0,0,%2\\n\" /* load and reserve */\r\n \"cmpw %0, %4\\n\" /* if load is not equal to */\r\n \"bne 2f\\n\" /* old, fail */\r\n \"stwcx. %3,0,%2\\n\" /* else store conditional */\r\n \"bne- 1b\\n\" /* retry if lost reservation */\r\n \"li %1,1\\n\" /* result = 1; */\r\n \"2:\\n\"\r\n : \"=&r\"(oldval), \"=&r\"(result)\r\n : \"r\"(addr), \"r\"(new_val), \"r\"(old), \"1\"(result)\r\n : \"memory\", \"cr0\");\r\n\r\n return result;\r\n}\r\n#endif\r\n\r\n#define AO_HAVE_compare_and_swap\r\n\r\nAO_INLINE int\r\nAO_compare_and_swap_acquire(volatile AO_t *addr, AO_t old, AO_t new_val) {\r\n int result = AO_compare_and_swap(addr, old, new_val);\r\n AO_lwsync();\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_compare_and_swap_acquire\r\n\r\nAO_INLINE int\r\nAO_compare_and_swap_release(volatile AO_t *addr, AO_t old, AO_t new_val) {\r\n AO_lwsync();\r\n return AO_compare_and_swap(addr, old, new_val);\r\n}\r\n\r\n#define AO_HAVE_compare_and_swap_release\r\n\r\nAO_INLINE int\r\nAO_compare_and_swap_full(volatile AO_t *addr, AO_t old, AO_t new_val) {\r\n AO_t result;\r\n AO_lwsync();\r\n result = AO_compare_and_swap(addr, old, new_val);\r\n AO_lwsync();\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_compare_and_swap_full\r\n\r\n#if defined(__powerpc64__) || defined(__ppc64__) || defined(__64BIT__)\r\n/* FIXME: Completely untested. */\r\n\r\nAO_INLINE AO_t\r\nAO_fetch_and_add(volatile AO_t *addr, AO_t incr) {\r\n AO_t oldval;\r\n AO_t newval;\r\n\r\n __asm__ __volatile__(\r\n \"1:ldarx %0,0,%2\\n\" /* load and reserve */\r\n \"add %1,%0,%3\\n\" /* increment */\r\n \"stdcx. %1,0,%2\\n\" /* store conditional */\r\n \"bne- 1b\\n\" /* retry if lost reservation */\r\n : \"=&r\"(oldval), \"=&r\"(newval)\r\n : \"r\"(addr), \"r\"(incr)\r\n : \"memory\", \"cr0\");\r\n\r\n return oldval;\r\n}\r\n\r\n#define AO_HAVE_fetch_and_add\r\n\r\n#else\r\n\r\nAO_INLINE AO_t\r\nAO_fetch_and_add(volatile AO_t *addr, AO_t incr) {\r\n AO_t oldval;\r\n AO_t newval;\r\n\r\n __asm__ __volatile__(\r\n \"1:lwarx %0,0,%2\\n\" /* load and reserve */\r\n \"add %1,%0,%3\\n\" /* increment */\r\n \"stwcx. %1,0,%2\\n\" /* store conditional */\r\n \"bne- 1b\\n\" /* retry if lost reservation */\r\n : \"=&r\"(oldval), \"=&r\"(newval)\r\n : \"r\"(addr), \"r\"(incr)\r\n : \"memory\", \"cr0\");\r\n\r\n return oldval;\r\n}\r\n\r\n#define AO_HAVE_fetch_and_add\r\n\r\n#endif\r\n\r\nAO_INLINE AO_t\r\nAO_fetch_and_add_acquire(volatile AO_t *addr, AO_t incr) {\r\n AO_t result = AO_fetch_and_add(addr, incr);\r\n AO_lwsync();\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_fetch_and_add_acquire\r\n\r\nAO_INLINE AO_t\r\nAO_fetch_and_add_release(volatile AO_t *addr, AO_t incr) {\r\n AO_lwsync();\r\n return AO_fetch_and_add(addr, incr);\r\n}\r\n\r\n#define AO_HAVE_fetch_and_add_release\r\n\r\nAO_INLINE AO_t\r\nAO_fetch_and_add_full(volatile AO_t *addr, AO_t incr) {\r\n AO_t result;\r\n AO_lwsync();\r\n result = AO_fetch_and_add(addr, incr);\r\n AO_lwsync();\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_fetch_and_add_full\r\n\r\n#if defined(__powerpc64__) || defined(__ppc64__) || defined(__64BIT__)\r\n#else\r\n# include \"../ao_t_is_int.h\"\r\n#endif\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/gcc/s390.h", "content": "/*\r\n * Copyright (c) 1991-1994 by Xerox Corporation. All rights reserved.\r\n * Copyright (c) 1996-1999 by Silicon Graphics. All rights reserved.\r\n * Copyright (c) 1999-2003 by Hewlett-Packard Company. All rights reserved.\r\n *\r\n *\r\n * THIS MATERIAL IS PROVIDED AS IS, WITH ABSOLUTELY NO WARRANTY EXPRESSED\r\n * OR IMPLIED. ANY USE IS AT YOUR OWN RISK.\r\n *\r\n * Permission is hereby granted to use or copy this program\r\n * for any purpose, provided the above notices are retained on all copies.\r\n * Permission to modify the code and to distribute modified code is granted,\r\n * provided the above notices are retained, and a notice that the code was\r\n * modified is included with the above copyright notice.\r\n *\r\n */\r\n\r\n/* FIXME: untested. */\r\n/* The relevant documentation appears to be at */\r\n/* http://publibz.boulder.ibm.com/epubs/pdf/dz9zr003.pdf */\r\n/* around page 5-96. Apparently: */\r\n/* - Memory references in general are atomic only for a single */\r\n/* byte. But it appears that the most common load/store */\r\n/* instructions also guarantee atomicity for aligned */\r\n/* operands of standard types. WE FOOLISHLY ASSUME that */\r\n/* compilers only generate those. If that turns out to be */\r\n/* wrong, we need inline assembly code for AO_load and */\r\n/* AO_store. */\r\n/* - A store followed by a load is unordered since the store */\r\n/* may be delayed. Otherwise everything is ordered. */\r\n/* - There is a hardware compare-and-swap (CS) instruction. */\r\n\r\n#include \"../ordered_except_wr.h\"\r\n#include \"../all_aligned_atomic_load_store.h\"\r\n\r\n#include \"../test_and_set_t_is_ao_t.h\"\r\n/* FIXME: Is there a way to do byte-sized test-and-set? */\r\n\r\n/* FIXME: AO_nop_full should probably be implemented directly. */\r\n/* It appears that certain BCR instructions have that effect. */\r\n/* Presumably they're cheaper than CS? */\r\n\r\nAO_INLINE AO_t AO_compare_and_swap_full(volatile AO_t *addr,\r\n AO_t old, AO_t new_val)\r\n{\r\n int retval;\r\n __asm__ __volatile__ (\r\n# ifndef __s390x__\r\n \" cs %1,%2,0(%3)\\n\"\r\n# else\r\n \" csg %1,%2,0(%3)\\n\"\r\n# endif\r\n \" ipm %0\\n\"\r\n \" srl %0,28\\n\"\r\n : \"=&d\" (retval), \"+d\" (old)\r\n : \"d\" (new_val), \"a\" (addr)\r\n : \"cc\", \"memory\");\r\n return retval == 0;\r\n}\r\n\r\n#define AO_HAVE_compare_and_swap_full\r\n\r\n/* FIXME: Add double-wide compare-and-swap for 32-bit executables. */\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/gcc/sh.h", "content": "/*\r\n * Copyright (c) 2009 by Takashi YOSHII. All rights reserved.\r\n *\r\n *\r\n * THIS MATERIAL IS PROVIDED AS IS, WITH ABSOLUTELY NO WARRANTY EXPRESSED\r\n * OR IMPLIED. ANY USE IS AT YOUR OWN RISK.\r\n *\r\n * Permission is hereby granted to use or copy this program\r\n * for any purpose, provided the above notices are retained on all copies.\r\n * Permission to modify the code and to distribute modified code is granted,\r\n * provided the above notices are retained, and a notice that the code was\r\n * modified is included with the above copyright notice.\r\n */\r\n\r\n#include \"../all_atomic_load_store.h\"\r\n#include \"../ordered.h\"\r\n\r\n/* sh has tas.b(byte) only */\r\n#include \"../test_and_set_t_is_char.h\"\r\n\r\nAO_INLINE AO_TS_VAL_t\r\nAO_test_and_set_full(volatile AO_TS_t *addr)\r\n{\r\n int oldval;\r\n __asm__ __volatile__(\r\n \"tas.b @%1; movt %0\"\r\n : \"=r\" (oldval)\r\n : \"r\" (addr)\r\n : \"t\", \"memory\");\r\n return oldval? AO_TS_CLEAR : AO_TS_SET;\r\n}\r\n#define AO_HAVE_test_and_set_full\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/gcc/sparc.h", "content": "/*\r\n * Copyright (c) 1991-1994 by Xerox Corporation. All rights reserved.\r\n * Copyright (c) 1996-1999 by Silicon Graphics. All rights reserved.\r\n * Copyright (c) 1999-2003 by Hewlett-Packard Company. All rights reserved.\r\n *\r\n *\r\n * THIS MATERIAL IS PROVIDED AS IS, WITH ABSOLUTELY NO WARRANTY EXPRESSED\r\n * OR IMPLIED. ANY USE IS AT YOUR OWN RISK.\r\n *\r\n * Permission is hereby granted to use or copy this program\r\n * for any purpose, provided the above notices are retained on all copies.\r\n * Permission to modify the code and to distribute modified code is granted,\r\n * provided the above notices are retained, and a notice that the code was\r\n * modified is included with the above copyright notice.\r\n *\r\n */\r\n\r\n/* FIXME. Very incomplete. No support for sparc64. */\r\n/* Non-ancient SPARCs provide compare-and-swap (casa). */\r\n/* We should make that available. */\r\n\r\n#include \"../all_atomic_load_store.h\"\r\n\r\n/* Real SPARC code uses TSO: */\r\n#include \"../ordered_except_wr.h\"\r\n\r\n/* Test_and_set location is just a byte. */\r\n#include \"../test_and_set_t_is_char.h\"\r\n\r\nAO_INLINE AO_TS_VAL_t\r\nAO_test_and_set_full(volatile AO_TS_t *addr) {\r\n AO_TS_VAL_t oldval;\r\n\r\n __asm__ __volatile__(\"ldstub %1,%0\"\r\n : \"=r\"(oldval), \"=m\"(*addr)\r\n : \"m\"(*addr) : \"memory\");\r\n return oldval;\r\n}\r\n\r\n#define AO_HAVE_test_and_set_full\r\n\r\n#ifndef AO_NO_SPARC_V9\r\n/* Returns nonzero if the comparison succeeded. */\r\nAO_INLINE int\r\nAO_compare_and_swap_full(volatile AO_t *addr, AO_t old, AO_t new_val) {\r\n char ret;\r\n __asm__ __volatile__ (\"membar #StoreLoad | #LoadLoad\\n\\t\"\r\n# if defined(__arch64__)\r\n \"casx [%2],%0,%1\\n\\t\"\r\n# else\r\n \"cas [%2],%0,%1\\n\\t\" /* 32-bit version */\r\n# endif\r\n \"membar #StoreLoad | #StoreStore\\n\\t\"\r\n \"cmp %0,%1\\n\\t\"\r\n \"be,a 0f\\n\\t\"\r\n \"mov 1,%0\\n\\t\"/* one insn after branch always executed */\r\n \"clr %0\\n\\t\"\r\n \"0:\\n\\t\"\r\n : \"=r\" (ret), \"+r\" (new_val)\r\n : \"r\" (addr), \"0\" (old)\r\n : \"memory\", \"cc\");\r\n return (int)ret;\r\n}\r\n\r\n#define AO_HAVE_compare_and_swap_full\r\n#endif /* AO_NO_SPARC_V9 */\r\n\r\n/* FIXME: This needs to be extended for SPARC v8 and v9. */\r\n/* SPARC V8 also has swap. V9 has CAS. */\r\n/* There are barriers like membar #LoadStore. */\r\n/* CASA (32-bit) and CASXA(64-bit) instructions were */\r\n/* added in V9. */\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/gcc/x86.h", "content": "/*\r\n * Copyright (c) 1991-1994 by Xerox Corporation. All rights reserved.\r\n * Copyright (c) 1996-1999 by Silicon Graphics. All rights reserved.\r\n * Copyright (c) 1999-2003 by Hewlett-Packard Company. All rights reserved.\r\n *\r\n *\r\n * THIS MATERIAL IS PROVIDED AS IS, WITH ABSOLUTELY NO WARRANTY EXPRESSED\r\n * OR IMPLIED. ANY USE IS AT YOUR OWN RISK.\r\n *\r\n * Permission is hereby granted to use or copy this program\r\n * for any purpose, provided the above notices are retained on all copies.\r\n * Permission to modify the code and to distribute modified code is granted,\r\n * provided the above notices are retained, and a notice that the code was\r\n * modified is included with the above copyright notice.\r\n *\r\n * Some of the machine specific code was borrowed from our GC distribution.\r\n */\r\n\r\n/* The following really assume we have a 486 or better. Unfortunately */\r\n/* gcc doesn't define a suitable feature test macro based on command */\r\n/* line options. */\r\n/* We should perhaps test dynamically. */\r\n\r\n#include \"../all_aligned_atomic_load_store.h\"\r\n\r\n/* Real X86 implementations, except for some old WinChips, appear */\r\n/* to enforce ordering between memory operations, EXCEPT that a later */\r\n/* read can pass earlier writes, presumably due to the visible */\r\n/* presence of store buffers. */\r\n/* We ignore both the WinChips, and the fact that the official specs */\r\n/* seem to be much weaker (and arguably too weak to be usable). */\r\n\r\n#include \"../ordered_except_wr.h\"\r\n\r\n#include \"../test_and_set_t_is_char.h\"\r\n\r\n#include \"../standard_ao_double_t.h\"\r\n\r\n#if defined(AO_USE_PENTIUM4_INSTRS)\r\nAO_INLINE void\r\nAO_nop_full(void)\r\n{\r\n __asm__ __volatile__(\"mfence\" : : : \"memory\");\r\n}\r\n\r\n#define AO_HAVE_nop_full\r\n\r\n#else\r\n\r\n/* We could use the cpuid instruction. But that seems to be slower */\r\n/* than the default implementation based on test_and_set_full. Thus */\r\n/* we omit that bit of misinformation here. */\r\n\r\n#endif\r\n\r\n/* As far as we can tell, the lfence and sfence instructions are not */\r\n/* currently needed or useful for cached memory accesses. */\r\n\r\n/* Really only works for 486 and later */\r\nAO_INLINE AO_t\r\nAO_fetch_and_add_full (volatile AO_t *p, AO_t incr)\r\n{\r\n AO_t result;\r\n\r\n __asm__ __volatile__ (\"lock; xaddl %0, %1\" :\r\n \"=r\" (result), \"=m\" (*p) : \"0\" (incr), \"m\" (*p)\r\n : \"memory\");\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_fetch_and_add_full\r\n\r\nAO_INLINE unsigned char\r\nAO_char_fetch_and_add_full (volatile unsigned char *p, unsigned char incr)\r\n{\r\n unsigned char result;\r\n\r\n __asm__ __volatile__ (\"lock; xaddb %0, %1\" :\r\n \"=q\" (result), \"=m\" (*p) : \"0\" (incr), \"m\" (*p)\r\n : \"memory\");\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_char_fetch_and_add_full\r\n\r\nAO_INLINE unsigned short\r\nAO_short_fetch_and_add_full (volatile unsigned short *p, unsigned short incr)\r\n{\r\n unsigned short result;\r\n\r\n __asm__ __volatile__ (\"lock; xaddw %0, %1\" :\r\n \"=r\" (result), \"=m\" (*p) : \"0\" (incr), \"m\" (*p)\r\n : \"memory\");\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_short_fetch_and_add_full\r\n\r\n/* Really only works for 486 and later */\r\nAO_INLINE void\r\nAO_or_full (volatile AO_t *p, AO_t incr)\r\n{\r\n __asm__ __volatile__ (\"lock; orl %1, %0\" :\r\n \"=m\" (*p) : \"r\" (incr), \"m\" (*p) : \"memory\");\r\n}\r\n\r\n#define AO_HAVE_or_full\r\n\r\nAO_INLINE AO_TS_VAL_t\r\nAO_test_and_set_full(volatile AO_TS_t *addr)\r\n{\r\n unsigned char oldval;\r\n /* Note: the \"xchg\" instruction does not need a \"lock\" prefix */\r\n __asm__ __volatile__(\"xchgb %0, %1\"\r\n : \"=q\"(oldval), \"=m\"(*addr)\r\n : \"0\"(0xff), \"m\"(*addr) : \"memory\");\r\n return (AO_TS_VAL_t)oldval;\r\n}\r\n\r\n#define AO_HAVE_test_and_set_full\r\n\r\n/* Returns nonzero if the comparison succeeded. */\r\nAO_INLINE int\r\nAO_compare_and_swap_full(volatile AO_t *addr, AO_t old, AO_t new_val)\r\n{\r\n# ifdef AO_USE_SYNC_CAS_BUILTIN\r\n return (int)__sync_bool_compare_and_swap(addr, old, new_val);\r\n# else\r\n char result;\r\n __asm__ __volatile__(\"lock; cmpxchgl %3, %0; setz %1\"\r\n : \"=m\" (*addr), \"=a\" (result)\r\n : \"m\" (*addr), \"r\" (new_val), \"a\" (old) : \"memory\");\r\n return (int)result;\r\n# endif\r\n}\r\n\r\n#define AO_HAVE_compare_and_swap_full\r\n\r\n/* Returns nonzero if the comparison succeeded. */\r\n/* Really requires at least a Pentium. */\r\nAO_INLINE int\r\nAO_compare_double_and_swap_double_full(volatile AO_double_t *addr,\r\n AO_t old_val1, AO_t old_val2,\r\n AO_t new_val1, AO_t new_val2)\r\n{\r\n char result;\r\n#if __PIC__\r\n /* If PIC is turned on, we can't use %ebx as it is reserved for the\r\n GOT pointer. We can save and restore %ebx because GCC won't be\r\n using it for anything else (such as any of the m operands) */\r\n __asm__ __volatile__(\"pushl %%ebx;\" /* save ebx used for PIC GOT ptr */\r\n \"movl %6,%%ebx;\" /* move new_val2 to %ebx */\r\n \"lock; cmpxchg8b %0; setz %1;\"\r\n \"pop %%ebx;\" /* restore %ebx */\r\n : \"=m\"(*addr), \"=a\"(result)\r\n : \"m\"(*addr), \"d\" (old_val2), \"a\" (old_val1),\r\n \"c\" (new_val2), \"m\" (new_val1) : \"memory\");\r\n#else\r\n /* We can't just do the same thing in non-PIC mode, because GCC\r\n * might be using %ebx as the memory operand. We could have ifdef'd\r\n * in a clobber, but there's no point doing the push/pop if we don't\r\n * have to. */\r\n __asm__ __volatile__(\"lock; cmpxchg8b %0; setz %1;\"\r\n : \"=m\"(*addr), \"=a\"(result)\r\n : \"m\"(*addr), \"d\" (old_val2), \"a\" (old_val1),\r\n \"c\" (new_val2), \"b\" (new_val1) : \"memory\");\r\n#endif\r\n return (int) result;\r\n}\r\n\r\n#define AO_HAVE_compare_double_and_swap_double_full\r\n\r\n#include \"../ao_t_is_int.h\"\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/gcc/x86_64.h", "content": "/*\r\n * Copyright (c) 1991-1994 by Xerox Corporation. All rights reserved.\r\n * Copyright (c) 1996-1999 by Silicon Graphics. All rights reserved.\r\n * Copyright (c) 1999-2003 by Hewlett-Packard Company. All rights reserved.\r\n *\r\n *\r\n * THIS MATERIAL IS PROVIDED AS IS, WITH ABSOLUTELY NO WARRANTY EXPRESSED\r\n * OR IMPLIED. ANY USE IS AT YOUR OWN RISK.\r\n *\r\n * Permission is hereby granted to use or copy this program\r\n * for any purpose, provided the above notices are retained on all copies.\r\n * Permission to modify the code and to distribute modified code is granted,\r\n * provided the above notices are retained, and a notice that the code was\r\n * modified is included with the above copyright notice.\r\n *\r\n * Some of the machine specific code was borrowed from our GC distribution.\r\n */\r\n\r\n#include \"../all_aligned_atomic_load_store.h\"\r\n\r\n/* Real X86 implementations appear */\r\n/* to enforce ordering between memory operations, EXCEPT that a later */\r\n/* read can pass earlier writes, presumably due to the visible */\r\n/* presence of store buffers. */\r\n/* We ignore the fact that the official specs */\r\n/* seem to be much weaker (and arguably too weak to be usable). */\r\n\r\n#include \"../ordered_except_wr.h\"\r\n\r\n#include \"../test_and_set_t_is_char.h\"\r\n\r\n#include \"../standard_ao_double_t.h\"\r\n\r\nAO_INLINE void\r\nAO_nop_full(void)\r\n{\r\n /* Note: \"mfence\" (SSE2) is supported on all x86_64/amd64 chips. */\r\n __asm__ __volatile__(\"mfence\" : : : \"memory\");\r\n}\r\n\r\n#define AO_HAVE_nop_full\r\n\r\n/* As far as we can tell, the lfence and sfence instructions are not */\r\n/* currently needed or useful for cached memory accesses. */\r\n\r\nAO_INLINE AO_t\r\nAO_fetch_and_add_full (volatile AO_t *p, AO_t incr)\r\n{\r\n AO_t result;\r\n\r\n __asm__ __volatile__ (\"lock; xaddq %0, %1\" :\r\n \"=r\" (result), \"=m\" (*p) : \"0\" (incr), \"m\" (*p)\r\n : \"memory\");\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_fetch_and_add_full\r\n\r\nAO_INLINE unsigned char\r\nAO_char_fetch_and_add_full (volatile unsigned char *p, unsigned char incr)\r\n{\r\n unsigned char result;\r\n\r\n __asm__ __volatile__ (\"lock; xaddb %0, %1\" :\r\n \"=q\" (result), \"=m\" (*p) : \"0\" (incr), \"m\" (*p)\r\n : \"memory\");\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_char_fetch_and_add_full\r\n\r\nAO_INLINE unsigned short\r\nAO_short_fetch_and_add_full (volatile unsigned short *p, unsigned short incr)\r\n{\r\n unsigned short result;\r\n\r\n __asm__ __volatile__ (\"lock; xaddw %0, %1\" :\r\n \"=r\" (result), \"=m\" (*p) : \"0\" (incr), \"m\" (*p)\r\n : \"memory\");\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_short_fetch_and_add_full\r\n\r\nAO_INLINE unsigned int\r\nAO_int_fetch_and_add_full (volatile unsigned int *p, unsigned int incr)\r\n{\r\n unsigned int result;\r\n\r\n __asm__ __volatile__ (\"lock; xaddl %0, %1\" :\r\n \"=r\" (result), \"=m\" (*p) : \"0\" (incr), \"m\" (*p)\r\n : \"memory\");\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_int_fetch_and_add_full\r\n\r\nAO_INLINE void\r\nAO_or_full (volatile AO_t *p, AO_t incr)\r\n{\r\n __asm__ __volatile__ (\"lock; orq %1, %0\" :\r\n \"=m\" (*p) : \"r\" (incr), \"m\" (*p) : \"memory\");\r\n}\r\n\r\n#define AO_HAVE_or_full\r\n\r\nAO_INLINE AO_TS_VAL_t\r\nAO_test_and_set_full(volatile AO_TS_t *addr)\r\n{\r\n unsigned char oldval;\r\n /* Note: the \"xchg\" instruction does not need a \"lock\" prefix */\r\n __asm__ __volatile__(\"xchgb %0, %1\"\r\n : \"=q\"(oldval), \"=m\"(*addr)\r\n : \"0\"(0xff), \"m\"(*addr) : \"memory\");\r\n return (AO_TS_VAL_t)oldval;\r\n}\r\n\r\n#define AO_HAVE_test_and_set_full\r\n\r\n/* Returns nonzero if the comparison succeeded. */\r\nAO_INLINE int\r\nAO_compare_and_swap_full(volatile AO_t *addr, AO_t old, AO_t new_val)\r\n{\r\n# ifdef AO_USE_SYNC_CAS_BUILTIN\r\n return (int)__sync_bool_compare_and_swap(addr, old, new_val);\r\n# else\r\n char result;\r\n __asm__ __volatile__(\"lock; cmpxchgq %3, %0; setz %1\"\r\n : \"=m\" (*addr), \"=a\" (result)\r\n : \"m\" (*addr), \"r\" (new_val), \"a\" (old) : \"memory\");\r\n return (int) result;\r\n# endif\r\n}\r\n\r\n#define AO_HAVE_compare_and_swap_full\r\n\r\n#ifdef AO_CMPXCHG16B_AVAILABLE\r\n/* NEC LE-IT: older AMD Opterons are missing this instruction.\r\n * On these machines SIGILL will be thrown.\r\n * Define AO_WEAK_DOUBLE_CAS_EMULATION to have an emulated\r\n * (lock based) version available */\r\n/* HB: Changed this to not define either by default. There are\r\n * enough machines and tool chains around on which cmpxchg16b\r\n * doesn't work. And the emulation is unsafe by our usual rules.\r\n * Hoewever both are clearly useful in certain cases.\r\n */\r\nAO_INLINE int\r\nAO_compare_double_and_swap_double_full(volatile AO_double_t *addr,\r\n AO_t old_val1, AO_t old_val2,\r\n AO_t new_val1, AO_t new_val2)\r\n{\r\n char result;\r\n __asm__ __volatile__(\"lock; cmpxchg16b %0; setz %1\"\r\n : \"=m\"(*addr), \"=a\"(result)\r\n : \"m\"(*addr), \"d\" (old_val2), \"a\" (old_val1),\r\n \"c\" (new_val2), \"b\" (new_val1) : \"memory\");\r\n return (int) result;\r\n}\r\n#define AO_HAVE_compare_double_and_swap_double_full\r\n#else\r\n/* this one provides spinlock based emulation of CAS implemented in */\r\n/* atomic_ops.c. We probably do not want to do this here, since it is */\r\n/* not atomic with respect to other kinds of updates of *addr. On the */\r\n/* other hand, this may be a useful facility on occasion. */\r\n#ifdef AO_WEAK_DOUBLE_CAS_EMULATION\r\nint AO_compare_double_and_swap_double_emulation(volatile AO_double_t *addr,\r\n AO_t old_val1, AO_t old_val2,\r\n AO_t new_val1, AO_t new_val2);\r\n\r\nAO_INLINE int\r\nAO_compare_double_and_swap_double_full(volatile AO_double_t *addr,\r\n AO_t old_val1, AO_t old_val2,\r\n AO_t new_val1, AO_t new_val2)\r\n{\r\n return AO_compare_double_and_swap_double_emulation(addr,\r\n old_val1, old_val2,\r\n new_val1, new_val2);\r\n}\r\n#define AO_HAVE_compare_double_and_swap_double_full\r\n#endif /* AO_WEAK_DOUBLE_CAS_EMULATION */\r\n#endif /* AO_CMPXCHG16B_AVAILABLE */\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/generic_pthread.h", "content": "/*\r\n * Copyright (c) 2003 by Hewlett-Packard Company. All rights reserved.\r\n *\r\n * Permission is hereby granted, free of charge, to any person obtaining a copy\r\n * of this software and associated documentation files (the \"Software\"), to deal\r\n * in the Software without restriction, including without limitation the rights\r\n * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\r\n * copies of the Software, and to permit persons to whom the Software is\r\n * furnished to do so, subject to the following conditions:\r\n *\r\n * The above copyright notice and this permission notice shall be included in\r\n * all copies or substantial portions of the Software.\r\n *\r\n * THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\r\n * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\r\n * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\r\n * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\r\n * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\r\n * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\r\n * SOFTWARE.\r\n */\r\n\r\n/* The following is useful primarily for debugging and documentation. */\r\n/* We define various atomic operations by acquiring a global pthread */\r\n/* lock. The resulting implementation will perform poorly, but should */\r\n/* be correct unless it is used from signal handlers. */\r\n/* We assume that all pthread operations act like full memory barriers. */\r\n/* (We believe that is the intent of the specification.) */\r\n\r\n#include \r\n\r\n#include \"test_and_set_t_is_ao_t.h\"\r\n /* This is not necessarily compatible with the native */\r\n /* implementation. But those can't be safely mixed anyway. */\r\n\r\n/* We define only the full barrier variants, and count on the */\r\n/* generalization section below to fill in the rest. */\r\nextern pthread_mutex_t AO_pt_lock;\r\n\r\nAO_INLINE void\r\nAO_nop_full(void)\r\n{\r\n pthread_mutex_lock(&AO_pt_lock);\r\n pthread_mutex_unlock(&AO_pt_lock);\r\n}\r\n\r\n#define AO_HAVE_nop_full\r\n\r\nAO_INLINE AO_t\r\nAO_load_full(const volatile AO_t *addr)\r\n{\r\n AO_t result;\r\n pthread_mutex_lock(&AO_pt_lock);\r\n result = *addr;\r\n pthread_mutex_unlock(&AO_pt_lock);\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_load_full\r\n\r\nAO_INLINE void\r\nAO_store_full(volatile AO_t *addr, AO_t val)\r\n{\r\n pthread_mutex_lock(&AO_pt_lock);\r\n *addr = val;\r\n pthread_mutex_unlock(&AO_pt_lock);\r\n}\r\n\r\n#define AO_HAVE_store_full\r\n\r\nAO_INLINE unsigned char\r\nAO_char_load_full(const volatile unsigned char *addr)\r\n{\r\n unsigned char result;\r\n pthread_mutex_lock(&AO_pt_lock);\r\n result = *addr;\r\n pthread_mutex_unlock(&AO_pt_lock);\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_char_load_full\r\n\r\nAO_INLINE void\r\nAO_char_store_full(volatile unsigned char *addr, unsigned char val)\r\n{\r\n pthread_mutex_lock(&AO_pt_lock);\r\n *addr = val;\r\n pthread_mutex_unlock(&AO_pt_lock);\r\n}\r\n\r\n#define AO_HAVE_char_store_full\r\n\r\nAO_INLINE unsigned short\r\nAO_short_load_full(const volatile unsigned short *addr)\r\n{\r\n unsigned short result;\r\n pthread_mutex_lock(&AO_pt_lock);\r\n result = *addr;\r\n pthread_mutex_unlock(&AO_pt_lock);\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_short_load_full\r\n\r\nAO_INLINE void\r\nAO_short_store_full(volatile unsigned short *addr, unsigned short val)\r\n{\r\n pthread_mutex_lock(&AO_pt_lock);\r\n *addr = val;\r\n pthread_mutex_unlock(&AO_pt_lock);\r\n}\r\n\r\n#define AO_HAVE_short_store_full\r\n\r\nAO_INLINE unsigned int\r\nAO_int_load_full(const volatile unsigned int *addr)\r\n{\r\n unsigned int result;\r\n pthread_mutex_lock(&AO_pt_lock);\r\n result = *addr;\r\n pthread_mutex_unlock(&AO_pt_lock);\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_int_load_full\r\n\r\nAO_INLINE void\r\nAO_int_store_full(volatile unsigned int *addr, unsigned int val)\r\n{\r\n pthread_mutex_lock(&AO_pt_lock);\r\n *addr = val;\r\n pthread_mutex_unlock(&AO_pt_lock);\r\n}\r\n\r\n#define AO_HAVE_int_store_full\r\n\r\nAO_INLINE AO_TS_VAL_t\r\nAO_test_and_set_full(volatile AO_TS_t *addr)\r\n{\r\n AO_TS_VAL_t result;\r\n pthread_mutex_lock(&AO_pt_lock);\r\n result = (AO_TS_VAL_t)(*addr);\r\n *addr = AO_TS_SET;\r\n pthread_mutex_unlock(&AO_pt_lock);\r\n assert(result == AO_TS_SET || result == AO_TS_CLEAR);\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_test_and_set_full\r\n\r\nAO_INLINE AO_t\r\nAO_fetch_and_add_full(volatile AO_t *p, AO_t incr)\r\n{\r\n AO_t tmp;\r\n\r\n pthread_mutex_lock(&AO_pt_lock);\r\n tmp = *p;\r\n *p = tmp + incr;\r\n pthread_mutex_unlock(&AO_pt_lock);\r\n return tmp;\r\n}\r\n\r\n#define AO_HAVE_fetch_and_add_full\r\n\r\nAO_INLINE unsigned char\r\nAO_char_fetch_and_add_full(volatile unsigned char *p, unsigned char incr)\r\n{\r\n unsigned char tmp;\r\n\r\n pthread_mutex_lock(&AO_pt_lock);\r\n tmp = *p;\r\n *p = tmp + incr;\r\n pthread_mutex_unlock(&AO_pt_lock);\r\n return tmp;\r\n}\r\n\r\n#define AO_HAVE_char_fetch_and_add_full\r\n\r\nAO_INLINE unsigned short\r\nAO_short_fetch_and_add_full(volatile unsigned short *p, unsigned short incr)\r\n{\r\n unsigned short tmp;\r\n\r\n pthread_mutex_lock(&AO_pt_lock);\r\n tmp = *p;\r\n *p = tmp + incr;\r\n pthread_mutex_unlock(&AO_pt_lock);\r\n return tmp;\r\n}\r\n\r\n#define AO_HAVE_short_fetch_and_add_full\r\n\r\nAO_INLINE unsigned int\r\nAO_int_fetch_and_add_full(volatile unsigned int *p, unsigned int incr)\r\n{\r\n unsigned int tmp;\r\n\r\n pthread_mutex_lock(&AO_pt_lock);\r\n tmp = *p;\r\n *p = tmp + incr;\r\n pthread_mutex_unlock(&AO_pt_lock);\r\n return tmp;\r\n}\r\n\r\n#define AO_HAVE_int_fetch_and_add_full\r\n\r\nAO_INLINE void\r\nAO_or_full(volatile AO_t *p, AO_t incr)\r\n{\r\n AO_t tmp;\r\n\r\n pthread_mutex_lock(&AO_pt_lock);\r\n tmp = *p;\r\n *p = (tmp | incr);\r\n pthread_mutex_unlock(&AO_pt_lock);\r\n}\r\n\r\n#define AO_HAVE_or_full\r\n\r\nAO_INLINE int\r\nAO_compare_and_swap_full(volatile AO_t *addr,\r\n AO_t old, AO_t new_val)\r\n{\r\n pthread_mutex_lock(&AO_pt_lock);\r\n if (*addr == old)\r\n {\r\n *addr = new_val;\r\n pthread_mutex_unlock(&AO_pt_lock);\r\n return 1;\r\n }\r\n else\r\n pthread_mutex_unlock(&AO_pt_lock);\r\n return 0;\r\n}\r\n\r\n#define AO_HAVE_compare_and_swap_full\r\n\r\n/* Unlike real architectures, we define both double-width CAS variants. */\r\n\r\ntypedef struct {\r\n AO_t AO_val1;\r\n AO_t AO_val2;\r\n} AO_double_t;\r\n\r\n#define AO_HAVE_double_t\r\n\r\nAO_INLINE int\r\nAO_compare_double_and_swap_double_full(volatile AO_double_t *addr,\r\n AO_t old1, AO_t old2,\r\n AO_t new1, AO_t new2)\r\n{\r\n pthread_mutex_lock(&AO_pt_lock);\r\n if (addr -> AO_val1 == old1 && addr -> AO_val2 == old2)\r\n {\r\n addr -> AO_val1 = new1;\r\n addr -> AO_val2 = new2;\r\n pthread_mutex_unlock(&AO_pt_lock);\r\n return 1;\r\n }\r\n else\r\n pthread_mutex_unlock(&AO_pt_lock);\r\n return 0;\r\n}\r\n\r\n#define AO_HAVE_compare_double_and_swap_double_full\r\n\r\nAO_INLINE int\r\nAO_compare_and_swap_double_full(volatile AO_double_t *addr,\r\n AO_t old1,\r\n AO_t new1, AO_t new2)\r\n{\r\n pthread_mutex_lock(&AO_pt_lock);\r\n if (addr -> AO_val1 == old1)\r\n {\r\n addr -> AO_val1 = new1;\r\n addr -> AO_val2 = new2;\r\n pthread_mutex_unlock(&AO_pt_lock);\r\n return 1;\r\n }\r\n else\r\n pthread_mutex_unlock(&AO_pt_lock);\r\n return 0;\r\n}\r\n\r\n#define AO_HAVE_compare_and_swap_double_full\r\n\r\n/* We can't use hardware loads and stores, since they don't */\r\n/* interact correctly with atomic updates. */\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/hpc/hppa.h", "content": "/*\r\n * Copyright (c) 2003 Hewlett-Packard Development Company, L.P.\r\n *\r\n * Permission is hereby granted, free of charge, to any person obtaining a copy\r\n * of this software and associated documentation files (the \"Software\"), to deal\r\n * in the Software without restriction, including without limitation the rights\r\n * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\r\n * copies of the Software, and to permit persons to whom the Software is\r\n * furnished to do so, subject to the following conditions:\r\n *\r\n * The above copyright notice and this permission notice shall be included in\r\n * all copies or substantial portions of the Software.\r\n *\r\n * THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\r\n * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\r\n * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\r\n * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\r\n * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\r\n * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\r\n * SOFTWARE.\r\n *\r\n * Derived from the corresponding header file for gcc.\r\n *\r\n */\r\n\r\n#include \"../atomic_load_store.h\"\r\n\r\n/* Some architecture set descriptions include special \"ordered\" memory */\r\n/* operations. As far as we can tell, no existing processors actually */\r\n/* require those. Nor does it appear likely that future processors */\r\n/* will. */\r\n/* FIXME: */\r\n/* The PA emulator on Itanium may obey weaker restrictions. */\r\n/* There should be a mode in which we don't assume sequential */\r\n/* consistency here. */\r\n#include \"../ordered.h\"\r\n\r\n#include \r\n\r\n/* GCC will not guarantee the alignment we need, use four lock words */\r\n/* and select the correctly aligned datum. See the glibc 2.3.2 */\r\n/* linuxthread port for the original implementation. */\r\nstruct AO_pa_clearable_loc {\r\n int data[4];\r\n};\r\n\r\n#undef AO_TS_INITIALIZER\r\n#define AO_TS_t struct AO_pa_clearable_loc\r\n#define AO_TS_INITIALIZER {1,1,1,1}\r\n/* Switch meaning of set and clear, since we only have an atomic clear */\r\n/* instruction. */\r\ntypedef enum {AO_PA_TS_set = 0, AO_PA_TS_clear = 1} AO_PA_TS_val;\r\n#define AO_TS_VAL_t AO_PA_TS_val\r\n#define AO_TS_CLEAR AO_PA_TS_clear\r\n#define AO_TS_SET AO_PA_TS_set\r\n\r\n/* The hppa only has one atomic read and modify memory operation, */\r\n/* load and clear, so hppa spinlocks must use zero to signify that */\r\n/* someone is holding the lock. The address used for the ldcw */\r\n/* semaphore must be 16-byte aligned. */\r\n\r\n#define __ldcw(a, ret) \\\r\n _LDCWX(0 /* index */, 0 /* s */, a /* base */, ret);\r\n\r\n/* Because malloc only guarantees 8-byte alignment for malloc'd data, */\r\n/* and GCC only guarantees 8-byte alignment for stack locals, we can't */\r\n/* be assured of 16-byte alignment for atomic lock data even if we */\r\n/* specify \"__attribute ((aligned(16)))\" in the type declaration. So, */\r\n/* we use a struct containing an array of four ints for the atomic lock */\r\n/* type and dynamically select the 16-byte aligned int from the array */\r\n/* for the semaphore. */\r\n#define __PA_LDCW_ALIGNMENT 16\r\n\r\n#define __ldcw_align(a, ret) { \\\r\n ret = (unsigned long) a; \\\r\n ret += __PA_LDCW_ALIGNMENT - 1; \\\r\n ret &= ~(__PA_LDCW_ALIGNMENT - 1); \\\r\n}\r\n\r\n/* Works on PA 1.1 and PA 2.0 systems */\r\nAO_INLINE AO_TS_VAL_t\r\nAO_test_and_set_full(volatile AO_TS_t * addr)\r\n{\r\n register unsigned int ret;\r\n register unsigned long a;\r\n __ldcw_align (addr, a);\r\n __ldcw (a, ret);\r\n return ret;\r\n}\r\n\r\nAO_INLINE void\r\nAO_pa_clear(volatile AO_TS_t * addr)\r\n{\r\n unsigned long a;\r\n __ldcw_align (addr,a);\r\n AO_compiler_barrier();\r\n *(volatile unsigned int *)a = 1;\r\n}\r\n#define AO_CLEAR(addr) AO_pa_clear(addr)\r\n\r\n#define AO_HAVE_test_and_set_full\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/hpc/ia64.h", "content": "/*\r\n * Copyright (c) 2003 Hewlett-Packard Development Company, L.P.\r\n *\r\n * Permission is hereby granted, free of charge, to any person obtaining a copy\r\n * of this software and associated documentation files (the \"Software\"), to deal\r\n * in the Software without restriction, including without limitation the rights\r\n * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\r\n * copies of the Software, and to permit persons to whom the Software is\r\n * furnished to do so, subject to the following conditions:\r\n *\r\n * The above copyright notice and this permission notice shall be included in\r\n * all copies or substantial portions of the Software.\r\n *\r\n * THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\r\n * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\r\n * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\r\n * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\r\n * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\r\n * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\r\n * SOFTWARE.\r\n */\r\n\r\n/*\r\n * This file specifies Itanimum primitives for use with the HP compiler\r\n * under HP/UX. We use intrinsics instead of the inline assembly code in the\r\n * gcc file.\r\n */\r\n\r\n#include \"../all_atomic_load_store.h\"\r\n\r\n#include \"../all_acquire_release_volatile.h\"\r\n\r\n#include \"../test_and_set_t_is_char.h\"\r\n\r\n#include \r\n\r\n#ifdef __LP64__\r\n# define AO_T_FASIZE _FASZ_D\r\n# define AO_T_SIZE _SZ_D\r\n#else\r\n# define AO_T_FASIZE _FASZ_W\r\n# define AO_T_SIZE _SZ_W\r\n#endif\r\n\r\nAO_INLINE void\r\nAO_nop_full(void)\r\n{\r\n _Asm_mf();\r\n}\r\n#define AO_HAVE_nop_full\r\n\r\nAO_INLINE AO_t\r\nAO_fetch_and_add1_acquire (volatile AO_t *p)\r\n{\r\n return _Asm_fetchadd(AO_T_FASIZE, _SEM_ACQ, p, 1,\r\n _LDHINT_NONE, _DOWN_MEM_FENCE);\r\n}\r\n#define AO_HAVE_fetch_and_add1_acquire\r\n\r\nAO_INLINE AO_t\r\nAO_fetch_and_add1_release (volatile AO_t *p)\r\n{\r\n return _Asm_fetchadd(AO_T_FASIZE, _SEM_REL, p, 1,\r\n _LDHINT_NONE, _UP_MEM_FENCE);\r\n}\r\n\r\n#define AO_HAVE_fetch_and_add1_release\r\n\r\nAO_INLINE AO_t\r\nAO_fetch_and_sub1_acquire (volatile AO_t *p)\r\n{\r\n return _Asm_fetchadd(AO_T_FASIZE, _SEM_ACQ, p, -1,\r\n _LDHINT_NONE, _DOWN_MEM_FENCE);\r\n}\r\n\r\n#define AO_HAVE_fetch_and_sub1_acquire\r\n\r\nAO_INLINE AO_t\r\nAO_fetch_and_sub1_release (volatile AO_t *p)\r\n{\r\n return _Asm_fetchadd(AO_T_FASIZE, _SEM_REL, p, -1,\r\n _LDHINT_NONE, _UP_MEM_FENCE);\r\n}\r\n\r\n#define AO_HAVE_fetch_and_sub1_release\r\n\r\nAO_INLINE int\r\nAO_compare_and_swap_acquire(volatile AO_t *addr,\r\n AO_t old, AO_t new_val)\r\n{\r\n AO_t oldval;\r\n\r\n _Asm_mov_to_ar(_AREG_CCV, old, _DOWN_MEM_FENCE);\r\n oldval = _Asm_cmpxchg(AO_T_SIZE, _SEM_ACQ, addr,\r\n new_val, _LDHINT_NONE, _DOWN_MEM_FENCE);\r\n return (oldval == old);\r\n}\r\n\r\n#define AO_HAVE_compare_and_swap_acquire\r\n\r\nAO_INLINE int\r\nAO_compare_and_swap_release(volatile AO_t *addr,\r\n AO_t old, AO_t new_val)\r\n{\r\n AO_t oldval;\r\n _Asm_mov_to_ar(_AREG_CCV, old, _UP_MEM_FENCE);\r\n oldval = _Asm_cmpxchg(AO_T_SIZE, _SEM_REL, addr,\r\n new_val, _LDHINT_NONE, _UP_MEM_FENCE);\r\n /* Hopefully the compiler knows not to reorder the above two? */\r\n return (oldval == old);\r\n}\r\n\r\n#define AO_HAVE_compare_and_swap_release\r\n\r\nAO_INLINE int\r\nAO_char_compare_and_swap_acquire(volatile unsigned char *addr,\r\n unsigned char old, unsigned char new_val)\r\n{\r\n unsigned char oldval;\r\n\r\n _Asm_mov_to_ar(_AREG_CCV, old, _DOWN_MEM_FENCE);\r\n oldval = _Asm_cmpxchg(_SZ_B, _SEM_ACQ, addr,\r\n new_val, _LDHINT_NONE, _DOWN_MEM_FENCE);\r\n return (oldval == old);\r\n}\r\n\r\n#define AO_HAVE_char_compare_and_swap_acquire\r\n\r\nAO_INLINE int\r\nAO_char_compare_and_swap_release(volatile unsigned char *addr,\r\n unsigned char old, unsigned char new_val)\r\n{\r\n unsigned char oldval;\r\n _Asm_mov_to_ar(_AREG_CCV, old, _UP_MEM_FENCE);\r\n oldval = _Asm_cmpxchg(_SZ_B, _SEM_REL, addr,\r\n new_val, _LDHINT_NONE, _UP_MEM_FENCE);\r\n /* Hopefully the compiler knows not to reorder the above two? */\r\n return (oldval == old);\r\n}\r\n\r\n#define AO_HAVE_char_compare_and_swap_release\r\n\r\nAO_INLINE int\r\nAO_short_compare_and_swap_acquire(volatile unsigned short *addr,\r\n unsigned short old, unsigned short new_val)\r\n{\r\n unsigned short oldval;\r\n\r\n _Asm_mov_to_ar(_AREG_CCV, old, _DOWN_MEM_FENCE);\r\n oldval = _Asm_cmpxchg(_SZ_B, _SEM_ACQ, addr,\r\n new_val, _LDHINT_NONE, _DOWN_MEM_FENCE);\r\n return (oldval == old);\r\n}\r\n\r\n#define AO_HAVE_short_compare_and_swap_acquire\r\n\r\nAO_INLINE int\r\nAO_short_compare_and_swap_release(volatile unsigned short *addr,\r\n unsigned short old, unsigned short new_val)\r\n{\r\n unsigned short oldval;\r\n _Asm_mov_to_ar(_AREG_CCV, old, _UP_MEM_FENCE);\r\n oldval = _Asm_cmpxchg(_SZ_B, _SEM_REL, addr,\r\n new_val, _LDHINT_NONE, _UP_MEM_FENCE);\r\n /* Hopefully the compiler knows not to reorder the above two? */\r\n return (oldval == old);\r\n}\r\n\r\n#define AO_HAVE_short_compare_and_swap_release\r\n\r\n#ifndef __LP64__\r\n# include \"../ao_t_is_int.h\"\r\n#endif\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/ibmc/powerpc.h", "content": "/* FIXME. This is only a placeholder for the AIX compiler. */\r\n/* It doesn't work. Please send a patch. */\r\n/* Memory model documented at http://www-106.ibm.com/developerworks/ */\r\n/* eserver/articles/archguide.html and (clearer) */\r\n/* http://www-106.ibm.com/developerworks/eserver/articles/powerpc.html. */\r\n/* There appears to be no implicit ordering between any kind of */\r\n/* independent memory references. */\r\n/* Architecture enforces some ordering based on control dependence. */\r\n/* I don't know if that could help. */\r\n/* Data-dependent loads are always ordered. */\r\n/* Based on the above references, eieio is intended for use on */\r\n/* uncached memory, which we don't support. It does not order loads */\r\n/* from cached memory. */\r\n/* Thanks to Maged Michael, Doug Lea, and Roger Hoover for helping to */\r\n/* track some of this down and correcting my misunderstandings. -HB */\r\n\r\n#include \"../all_aligned_atomic_load_store.h\"\r\n\r\nvoid AO_sync(void);\r\n#pragma mc_func AO_sync { \"7c0004ac\" }\r\n\r\n#ifdef __NO_LWSYNC__\r\n# define AO_lwsync AO_sync\r\n#else\r\n void AO_lwsync(void);\r\n#pragma mc_func AO_lwsync { \"7c2004ac\" }\r\n#endif\r\n\r\n#define AO_nop_write() AO_lwsync()\r\n#define AO_HAVE_nop_write\r\n\r\n#define AO_nop_read() AO_lwsync()\r\n#define AO_HAVE_nop_read\r\n\r\n/* We explicitly specify load_acquire and store_release, since these */\r\n/* rely on the fact that lwsync is also a LoadStore barrier. */\r\nAO_INLINE AO_t\r\nAO_load_acquire(const volatile AO_t *addr)\r\n{\r\n AO_t result = *addr;\r\n AO_lwsync();\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_load_acquire\r\n\r\nAO_INLINE void\r\nAO_store_release(volatile AO_t *addr, AO_t value)\r\n{\r\n AO_lwsync();\r\n *addr = value;\r\n}\r\n\r\n#define AO_HAVE_load_acquire\r\n\r\n/* This is similar to the code in the garbage collector. Deleting */\r\n/* this and having it synthesized from compare_and_swap would probably */\r\n/* only cost us a load immediate instruction. */\r\n/*AO_INLINE AO_TS_VAL_t\r\nAO_test_and_set(volatile AO_TS_t *addr) {\r\n# error FIXME Implement me\r\n}\r\n\r\n#define AO_HAVE_test_and_set*/\r\n\r\nAO_INLINE AO_TS_VAL_t\r\nAO_test_and_set_acquire(volatile AO_TS_t *addr) {\r\n AO_TS_VAL_t result = AO_test_and_set(addr);\r\n AO_lwsync();\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_test_and_set_acquire\r\n\r\nAO_INLINE AO_TS_VAL_t\r\nAO_test_and_set_release(volatile AO_TS_t *addr) {\r\n AO_lwsync();\r\n return AO_test_and_set(addr);\r\n}\r\n\r\n#define AO_HAVE_test_and_set_release\r\n\r\nAO_INLINE AO_TS_VAL_t\r\nAO_test_and_set_full(volatile AO_TS_t *addr) {\r\n AO_TS_VAL_t result;\r\n AO_lwsync();\r\n result = AO_test_and_set(addr);\r\n AO_lwsync();\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_test_and_set_full\r\n\r\n/*AO_INLINE AO_t\r\nAO_compare_and_swap(volatile AO_t *addr, AO_t old, AO_t new_val) {\r\n# error FIXME Implement me\r\n}\r\n\r\n#define AO_HAVE_compare_and_swap*/\r\n\r\nAO_INLINE AO_t\r\nAO_compare_and_swap_acquire(volatile AO_t *addr, AO_t old, AO_t new_val) {\r\n AO_t result = AO_compare_and_swap(addr, old, new_val);\r\n AO_lwsync();\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_compare_and_swap_acquire\r\n\r\nAO_INLINE AO_t\r\nAO_compare_and_swap_release(volatile AO_t *addr, AO_t old, AO_t new_val) {\r\n AO_lwsync();\r\n return AO_compare_and_swap(addr, old, new_val);\r\n}\r\n\r\n#define AO_HAVE_compare_and_swap_release\r\n\r\nAO_INLINE AO_t\r\nAO_compare_and_swap_full(volatile AO_t *addr, AO_t old, AO_t new_val) {\r\n AO_t result;\r\n AO_lwsync();\r\n result = AO_compare_and_swap(addr, old, new_val);\r\n AO_lwsync();\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_compare_and_swap_full\r\n\r\n/* FIXME: We should also implement fetch_and_add and or primitives */\r\n/* directly. */\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/icc/ia64.h", "content": "/*\r\n * Copyright (c) 2003 by Hewlett-Packard Company. All rights reserved.\r\n *\r\n * Permission is hereby granted, free of charge, to any person obtaining a copy\r\n * of this software and associated documentation files (the \"Software\"), to deal\r\n * in the Software without restriction, including without limitation the rights\r\n * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\r\n * copies of the Software, and to permit persons to whom the Software is\r\n * furnished to do so, subject to the following conditions:\r\n *\r\n * The above copyright notice and this permission notice shall be included in\r\n * all copies or substantial portions of the Software.\r\n *\r\n * THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\r\n * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\r\n * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\r\n * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\r\n * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\r\n * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\r\n * SOFTWARE.\r\n */\r\n\r\n/*\r\n * This file specifies Itanimum primitives for use with the Intel (ecc)\r\n * compiler. We use intrinsics instead of the inline assembly code in the\r\n * gcc file.\r\n */\r\n\r\n#include \"../all_atomic_load_store.h\"\r\n\r\n#include \"../test_and_set_t_is_char.h\"\r\n\r\n#include \r\n\r\n/* The acquire release semantics of volatile can be turned off. And volatile */\r\n/* operations in icc9 don't imply ordering with respect to other nonvolatile */\r\n/* operations. */\r\n\r\n#define AO_INTEL_PTR_t void *\r\n\r\nAO_INLINE AO_t\r\nAO_load_acquire(const volatile AO_t *p)\r\n{\r\n return (AO_t)(__ld8_acq((AO_INTEL_PTR_t)p));\r\n}\r\n#define AO_HAVE_load_acquire\r\n\r\nAO_INLINE void\r\nAO_store_release(volatile AO_t *p, AO_t val)\r\n{\r\n __st8_rel((AO_INTEL_PTR_t)p, (__int64)val);\r\n}\r\n#define AO_HAVE_store_release\r\n\r\nAO_INLINE unsigned char\r\nAO_char_load_acquire(const volatile unsigned char *p)\r\n{\r\n /* A normal volatile load generates an ld.acq */\r\n return (__ld1_acq((AO_INTEL_PTR_t)p));\r\n}\r\n#define AO_HAVE_char_load_acquire\r\n\r\nAO_INLINE void\r\nAO_char_store_release(volatile unsigned char *p, unsigned char val)\r\n{\r\n __st1_rel((AO_INTEL_PTR_t)p, val);\r\n}\r\n#define AO_HAVE_char_store_release\r\n\r\nAO_INLINE unsigned short\r\nAO_short_load_acquire(const volatile unsigned short *p)\r\n{\r\n /* A normal volatile load generates an ld.acq */\r\n return (__ld2_acq((AO_INTEL_PTR_t)p));\r\n}\r\n#define AO_HAVE_short_load_acquire\r\n\r\nAO_INLINE void\r\nAO_short_store_release(volatile unsigned short *p, unsigned short val)\r\n{\r\n __st2_rel((AO_INTEL_PTR_t)p, val);\r\n}\r\n#define AO_HAVE_short_store_release\r\n\r\nAO_INLINE unsigned int\r\nAO_int_load_acquire(const volatile unsigned int *p)\r\n{\r\n /* A normal volatile load generates an ld.acq */\r\n return (__ld4_acq((AO_INTEL_PTR_t)p));\r\n}\r\n#define AO_HAVE_int_load_acquire\r\n\r\nAO_INLINE void\r\nAO_int_store_release(volatile unsigned int *p, unsigned int val)\r\n{\r\n __st4_rel((AO_INTEL_PTR_t)p, val);\r\n}\r\n#define AO_HAVE_int_store_release\r\n\r\nAO_INLINE void\r\nAO_nop_full(void)\r\n{\r\n __mf();\r\n}\r\n#define AO_HAVE_nop_full\r\n\r\nAO_INLINE AO_t\r\nAO_fetch_and_add1_acquire (volatile AO_t *p)\r\n{\r\n return __fetchadd8_acq((unsigned __int64 *)p, 1);\r\n}\r\n#define AO_HAVE_fetch_and_add1_acquire\r\n\r\nAO_INLINE AO_t\r\nAO_fetch_and_add1_release (volatile AO_t *p)\r\n{\r\n return __fetchadd8_rel((unsigned __int64 *)p, 1);\r\n}\r\n\r\n#define AO_HAVE_fetch_and_add1_release\r\n\r\nAO_INLINE AO_t\r\nAO_fetch_and_sub1_acquire (volatile AO_t *p)\r\n{\r\n return __fetchadd8_acq((unsigned __int64 *)p, -1);\r\n}\r\n\r\n#define AO_HAVE_fetch_and_sub1_acquire\r\n\r\nAO_INLINE AO_t\r\nAO_fetch_and_sub1_release (volatile AO_t *p)\r\n{\r\n return __fetchadd8_rel((unsigned __int64 *)p, -1);\r\n}\r\n\r\n#define AO_HAVE_fetch_and_sub1_release\r\n\r\nAO_INLINE int\r\nAO_compare_and_swap_acquire(volatile AO_t *addr,\r\n AO_t old, AO_t new_val)\r\n{\r\n AO_t oldval;\r\n oldval = _InterlockedCompareExchange64_acq(addr, new_val, old);\r\n return (oldval == old);\r\n}\r\n\r\n#define AO_HAVE_compare_and_swap_acquire\r\n\r\nAO_INLINE int\r\nAO_compare_and_swap_release(volatile AO_t *addr,\r\n AO_t old, AO_t new_val)\r\n{\r\n AO_t oldval;\r\n oldval = _InterlockedCompareExchange64_rel(addr, new_val, old);\r\n return (oldval == old);\r\n}\r\n\r\n#define AO_HAVE_compare_and_swap_release\r\n\r\nAO_INLINE int\r\nAO_char_compare_and_swap_acquire(volatile unsigned char *addr,\r\n unsigned char old, unsigned char new_val)\r\n{\r\n unsigned char oldval;\r\n oldval = _InterlockedCompareExchange8_acq(addr, new_val, old);\r\n return (oldval == old);\r\n}\r\n\r\n#define AO_HAVE_char_compare_and_swap_acquire\r\n\r\nAO_INLINE int\r\nAO_char_compare_and_swap_release(volatile unsigned char *addr,\r\n unsigned char old, unsigned char new_val)\r\n{\r\n unsigned char oldval;\r\n oldval = _InterlockedCompareExchange8_rel(addr, new_val, old);\r\n return (oldval == old);\r\n}\r\n\r\n#define AO_HAVE_char_compare_and_swap_release\r\n\r\nAO_INLINE int\r\nAO_short_compare_and_swap_acquire(volatile unsigned short *addr,\r\n unsigned short old, unsigned short new_val)\r\n{\r\n unsigned short oldval;\r\n oldval = _InterlockedCompareExchange16_acq(addr, new_val, old);\r\n return (oldval == old);\r\n}\r\n\r\n#define AO_HAVE_short_compare_and_swap_acquire\r\n\r\nAO_INLINE int\r\nAO_short_compare_and_swap_release(volatile unsigned short *addr,\r\n unsigned short old, unsigned short new_val)\r\n{\r\n unsigned short oldval;\r\n oldval = _InterlockedCompareExchange16_rel(addr, new_val, old);\r\n return (oldval == old);\r\n}\r\n\r\n#define AO_HAVE_short_compare_and_swap_release\r\n\r\nAO_INLINE int\r\nAO_int_compare_and_swap_acquire(volatile unsigned int *addr,\r\n unsigned int old, unsigned int new_val)\r\n{\r\n unsigned int oldval;\r\n oldval = _InterlockedCompareExchange_acq(addr, new_val, old);\r\n return (oldval == old);\r\n}\r\n\r\n#define AO_HAVE_int_compare_and_swap_acquire\r\n\r\nAO_INLINE int\r\nAO_int_compare_and_swap_release(volatile unsigned int *addr,\r\n unsigned int old, unsigned int new_val)\r\n{\r\n unsigned int oldval;\r\n oldval = _InterlockedCompareExchange_rel(addr, new_val, old);\r\n return (oldval == old);\r\n}\r\n\r\n#define AO_HAVE_int_compare_and_swap_release\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/int_acquire_release_volatile.h", "content": "/*\r\n * Copyright (c) 2003-2004 Hewlett-Packard Development Company, L.P.\r\n *\r\n * Permission is hereby granted, free of charge, to any person obtaining a copy\r\n * of this software and associated documentation files (the \"Software\"), to deal\r\n * in the Software without restriction, including without limitation the rights\r\n * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\r\n * copies of the Software, and to permit persons to whom the Software is\r\n * furnished to do so, subject to the following conditions:\r\n *\r\n * The above copyright notice and this permission notice shall be included in\r\n * all copies or substantial portions of the Software.\r\n *\r\n * THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\r\n * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\r\n * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\r\n * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\r\n * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\r\n * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\r\n * SOFTWARE.\r\n */\r\n\r\n/*\r\n * This file adds definitions appropriate for environments in which an unsigned\r\n * int volatile load has acquire semantics, and an unsigned short volatile\r\n * store has release semantics. This is true with the standard Itanium ABI.\r\n */\r\n#if !defined(AO_GCC_BARRIER)\r\n# if defined(__GNUC__)\r\n# define AO_GCC_BARRIER() AO_compiler_barrier()\r\n# else\r\n# define AO_GCC_BARRIER()\r\n# endif\r\n#endif\r\n\r\nAO_INLINE unsigned int\r\nAO_int_load_acquire(const volatile unsigned int *p)\r\n{\r\n unsigned int result = *p;\r\n /* A normal volatile load generates an ld.acq */\r\n AO_GCC_BARRIER();\r\n return result;\r\n}\r\n#define AO_HAVE_int_load_acquire\r\n\r\nAO_INLINE void\r\nAO_int_store_release(volatile unsigned int *p, unsigned int val)\r\n{\r\n AO_GCC_BARRIER();\r\n /* A normal volatile store generates an st.rel */\r\n *p = val;\r\n}\r\n#define AO_HAVE_int_store_release\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/int_aligned_atomic_load_store.h", "content": "/*\r\n * Copyright (c) 2004 Hewlett-Packard Development Company, L.P.\r\n *\r\n * Permission is hereby granted, free of charge, to any person obtaining a copy\r\n * of this software and associated documentation files (the \"Software\"), to deal\r\n * in the Software without restriction, including without limitation the rights\r\n * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\r\n * copies of the Software, and to permit persons to whom the Software is\r\n * furnished to do so, subject to the following conditions:\r\n *\r\n * The above copyright notice and this permission notice shall be included in\r\n * all copies or substantial portions of the Software.\r\n *\r\n * THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\r\n * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\r\n * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\r\n * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\r\n * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\r\n * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\r\n * SOFTWARE.\r\n */\r\n\r\n/*\r\n * Definitions for architectures on which loads and stores of unsigned int are\r\n * atomic for all legal alignments.\r\n */\r\n\r\nAO_INLINE unsigned int\r\nAO_int_load(const volatile unsigned int *addr)\r\n{\r\n assert(((size_t)addr & (sizeof(unsigned int) - 1)) == 0);\r\n /* Cast away the volatile for architectures like IA64 where */\r\n /* volatile adds barrier semantics. */\r\n return (*(unsigned int *)addr);\r\n}\r\n\r\n#define AO_HAVE_int_load\r\n\r\nAO_INLINE void\r\nAO_int_store(volatile unsigned int *addr, unsigned int new_val)\r\n{\r\n assert(((size_t)addr & (sizeof(unsigned int) - 1)) == 0);\r\n (*(unsigned int *)addr) = new_val;\r\n}\r\n\r\n#define AO_HAVE_int_store\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/int_atomic_load_store.h", "content": "/*\r\n * Copyright (c) 2003 by Hewlett-Packard Company. All rights reserved.\r\n *\r\n * Permission is hereby granted, free of charge, to any person obtaining a copy\r\n * of this software and associated documentation files (the \"Software\"), to deal\r\n * in the Software without restriction, including without limitation the rights\r\n * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\r\n * copies of the Software, and to permit persons to whom the Software is\r\n * furnished to do so, subject to the following conditions:\r\n *\r\n * The above copyright notice and this permission notice shall be included in\r\n * all copies or substantial portions of the Software.\r\n *\r\n * THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\r\n * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\r\n * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\r\n * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\r\n * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\r\n * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\r\n * SOFTWARE.\r\n */\r\n\r\n/*\r\n * Definitions for architectures on which loads and stores of unsigned int are\r\n * atomic for all legal alignments.\r\n */\r\n\r\nAO_INLINE unsigned int\r\nAO_int_load(const volatile unsigned int *addr)\r\n{\r\n /* Cast away the volatile for architectures like IA64 where */\r\n /* volatile adds barrier semantics. */\r\n return (*(const unsigned int *)addr);\r\n}\r\n\r\n#define AO_HAVE_int_load\r\n\r\nAO_INLINE void\r\nAO_int_store(volatile unsigned int *addr, unsigned int new_val)\r\n{\r\n (*(unsigned int *)addr) = new_val;\r\n}\r\n\r\n#define AO_HAVE_int_store\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/msftc/arm.h", "content": "/*\r\n * Copyright (c) 2003 Hewlett-Packard Development Company, L.P.\r\n *\r\n * Permission is hereby granted, free of charge, to any person obtaining a copy\r\n * of this software and associated documentation files (the \"Software\"), to deal\r\n * in the Software without restriction, including without limitation the rights\r\n * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\r\n * copies of the Software, and to permit persons to whom the Software is\r\n * furnished to do so, subject to the following conditions:\r\n *\r\n * The above copyright notice and this permission notice shall be included in\r\n * all copies or substantial portions of the Software.\r\n *\r\n * THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\r\n * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\r\n * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\r\n * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\r\n * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\r\n * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\r\n * SOFTWARE.\r\n */\r\n\r\n#include \"../read_ordered.h\"\r\n\r\n#ifndef AO_ASSUME_WINDOWS98\r\n /* CAS is always available */\r\n# define AO_ASSUME_WINDOWS98\r\n#endif\r\n#include \"common32_defs.h\"\r\n/* FIXME: Do _InterlockedOps really have a full memory barrier? */\r\n/* (MSDN WinCE docs say nothing about it.) */\r\n\r\n#if _M_ARM >= 6\r\n/* ARMv6 is the first architecture providing support for simple LL/SC. */\r\n\r\n#include \"../standard_ao_double_t.h\"\r\n\r\n/* If only a single processor is used, we can define AO_UNIPROCESSOR */\r\n/* and do not need to access CP15 for ensuring a DMB at all. */\r\n#ifdef AO_UNIPROCESSOR\r\n AO_INLINE void AO_nop_full(void) {}\r\n# define AO_HAVE_nop_full\r\n#else\r\n/* AO_nop_full() is emulated using AO_test_and_set_full(). */\r\n#endif\r\n\r\n#include \"../test_and_set_t_is_ao_t.h\"\r\n/* AO_test_and_set() is emulated using CAS. */\r\n\r\nAO_INLINE AO_t\r\nAO_load(const volatile AO_t *addr)\r\n{\r\n /* Cast away the volatile in case it adds fence semantics */\r\n return (*(const AO_t *)addr);\r\n}\r\n#define AO_HAVE_load\r\n\r\nAO_INLINE void\r\nAO_store_full(volatile AO_t *addr, AO_t value)\r\n{\r\n /* Emulate atomic store using CAS. */\r\n AO_t old = AO_load(addr);\r\n AO_t current;\r\n# ifdef AO_OLD_STYLE_INTERLOCKED_COMPARE_EXCHANGE\r\n while ((current = (AO_t)_InterlockedCompareExchange(\r\n (PVOID AO_INTERLOCKED_VOLATILE *)addr,\r\n (PVOID)value, (PVOID)old)) != old)\r\n old = current;\r\n# else\r\n while ((current = (AO_t)_InterlockedCompareExchange(\r\n (LONG AO_INTERLOCKED_VOLATILE *)addr,\r\n (LONG)value, (LONG)old)) != old)\r\n old = current;\r\n# endif\r\n}\r\n#define AO_HAVE_store_full\r\n\r\n/* FIXME: implement AO_compare_double_and_swap_double() */\r\n\r\n#else /* _M_ARM < 6 */\r\n\r\n/* Some slide set, if it has been red correctly, claims that Loads */\r\n/* followed by either a Load or a Store are ordered, but nothing */\r\n/* else is. It appears that SWP is the only simple memory barrier. */\r\n#include \"../all_atomic_load_store.h\"\r\n\r\n#include \"../test_and_set_t_is_ao_t.h\"\r\n/* AO_test_and_set_full() is emulated using CAS. */\r\n\r\n#endif /* _M_ARM < 6 */\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/msftc/common32_defs.h", "content": "/*\r\n * Copyright (c) 2003 Hewlett-Packard Development Company, L.P.\r\n *\r\n * Permission is hereby granted, free of charge, to any person obtaining a copy\r\n * of this software and associated documentation files (the \"Software\"), to deal\r\n * in the Software without restriction, including without limitation the rights\r\n * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\r\n * copies of the Software, and to permit persons to whom the Software is\r\n * furnished to do so, subject to the following conditions:\r\n *\r\n * The above copyright notice and this permission notice shall be included in\r\n * all copies or substantial portions of the Software.\r\n *\r\n * THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\r\n * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\r\n * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\r\n * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\r\n * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\r\n * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\r\n * SOFTWARE.\r\n */\r\n\r\n/* This file contains AO primitives based on VC++ built-in intrinsic */\r\n/* functions commonly available across 32-bit architectures. */\r\n\r\n/* This file should be included from arch-specific header files. */\r\n/* Define AO_USE_INTERLOCKED_INTRINSICS if _Interlocked primitives */\r\n/* (used below) are available as intrinsic ones for a target arch */\r\n/* (otherwise \"Interlocked\" functions family is used instead). */\r\n/* Define AO_ASSUME_WINDOWS98 if CAS is available. */\r\n\r\n#include \r\n /* Seems like over-kill, but that's what MSDN recommends. */\r\n /* And apparently winbase.h is not always self-contained. */\r\n\r\n#if _MSC_VER < 1310 || !defined(AO_USE_INTERLOCKED_INTRINSICS)\r\n\r\n# define _InterlockedIncrement InterlockedIncrement\r\n# define _InterlockedDecrement InterlockedDecrement\r\n# define _InterlockedExchange InterlockedExchange\r\n# define _InterlockedExchangeAdd InterlockedExchangeAdd\r\n# define _InterlockedCompareExchange InterlockedCompareExchange\r\n\r\n# define AO_INTERLOCKED_VOLATILE /**/\r\n\r\n#else /* elif _MSC_VER >= 1310 */\r\n\r\n# if _MSC_VER >= 1400\r\n# ifndef _WIN32_WCE\r\n# include \r\n# endif\r\n\r\n# pragma intrinsic (_ReadWriteBarrier)\r\n\r\n# else /* elif _MSC_VER < 1400 */\r\n# ifdef __cplusplus\r\n extern \"C\" {\r\n# endif\r\n LONG __cdecl _InterlockedIncrement(LONG volatile *);\r\n LONG __cdecl _InterlockedDecrement(LONG volatile *);\r\n LONG __cdecl _InterlockedExchangeAdd(LONG volatile *, LONG);\r\n LONG __cdecl _InterlockedExchange(LONG volatile *, LONG);\r\n LONG __cdecl _InterlockedCompareExchange(LONG volatile *,\r\n LONG /* Exchange */, LONG /* Comp */);\r\n# ifdef __cplusplus\r\n }\r\n# endif\r\n# endif /* _MSC_VER < 1400 */\r\n\r\n# pragma intrinsic (_InterlockedIncrement)\r\n# pragma intrinsic (_InterlockedDecrement)\r\n# pragma intrinsic (_InterlockedExchange)\r\n# pragma intrinsic (_InterlockedExchangeAdd)\r\n# pragma intrinsic (_InterlockedCompareExchange)\r\n\r\n# define AO_INTERLOCKED_VOLATILE volatile\r\n\r\n#endif /* _MSC_VER >= 1310 */\r\n\r\nAO_INLINE AO_t\r\nAO_fetch_and_add_full(volatile AO_t *p, AO_t incr)\r\n{\r\n return _InterlockedExchangeAdd((LONG AO_INTERLOCKED_VOLATILE *)p,\r\n (LONG)incr);\r\n}\r\n\r\n#define AO_HAVE_fetch_and_add_full\r\n\r\nAO_INLINE AO_t\r\nAO_fetch_and_add1_full(volatile AO_t *p)\r\n{\r\n return _InterlockedIncrement((LONG AO_INTERLOCKED_VOLATILE *)p) - 1;\r\n}\r\n\r\n#define AO_HAVE_fetch_and_add1_full\r\n\r\nAO_INLINE AO_t\r\nAO_fetch_and_sub1_full(volatile AO_t *p)\r\n{\r\n return _InterlockedDecrement((LONG AO_INTERLOCKED_VOLATILE *)p) + 1;\r\n}\r\n\r\n#define AO_HAVE_fetch_and_sub1_full\r\n\r\n#ifdef AO_ASSUME_WINDOWS98\r\n/* Returns nonzero if the comparison succeeded. */\r\nAO_INLINE int\r\nAO_compare_and_swap_full(volatile AO_t *addr, AO_t old, AO_t new_val)\r\n{\r\n# ifdef AO_OLD_STYLE_INTERLOCKED_COMPARE_EXCHANGE\r\n return _InterlockedCompareExchange((PVOID AO_INTERLOCKED_VOLATILE *)addr,\r\n (PVOID)new_val, (PVOID)old)\r\n == (PVOID)old;\r\n# else\r\n return _InterlockedCompareExchange((LONG AO_INTERLOCKED_VOLATILE *)addr,\r\n (LONG)new_val, (LONG)old)\r\n == (LONG)old;\r\n# endif\r\n}\r\n\r\n# define AO_HAVE_compare_and_swap_full\r\n#endif /* AO_ASSUME_WINDOWS98 */\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/msftc/x86.h", "content": "/*\r\n * Copyright (c) 2003 Hewlett-Packard Development Company, L.P.\r\n *\r\n * Permission is hereby granted, free of charge, to any person obtaining a copy\r\n * of this software and associated documentation files (the \"Software\"), to deal\r\n * in the Software without restriction, including without limitation the rights\r\n * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\r\n * copies of the Software, and to permit persons to whom the Software is\r\n * furnished to do so, subject to the following conditions:\r\n *\r\n * The above copyright notice and this permission notice shall be included in\r\n * all copies or substantial portions of the Software.\r\n *\r\n * THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\r\n * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\r\n * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\r\n * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\r\n * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\r\n * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\r\n * SOFTWARE.\r\n */\r\n\r\n/* If AO_ASSUME_WINDOWS98 is defined, we assume Windows 98 or newer. */\r\n/* If AO_ASSUME_VISTA is defined, we assume Windows Server 2003, Vista */\r\n/* or later. */\r\n\r\n#include \"../all_aligned_atomic_load_store.h\"\r\n\r\n/* Real X86 implementations, except for some old WinChips, appear */\r\n/* to enforce ordering between memory operations, EXCEPT that a later */\r\n/* read can pass earlier writes, presumably due to the visible */\r\n/* presence of store buffers. */\r\n/* We ignore both the WinChips, and the fact that the official specs */\r\n/* seem to be much weaker (and arguably too weak to be usable). */\r\n\r\n#include \"../ordered_except_wr.h\"\r\n\r\n#include \"../test_and_set_t_is_char.h\"\r\n\r\n#ifndef AO_USE_INTERLOCKED_INTRINSICS\r\n /* _Interlocked primitives (Inc, Dec, Xchg, Add) are always available */\r\n# define AO_USE_INTERLOCKED_INTRINSICS\r\n#endif\r\n#include \"common32_defs.h\"\r\n\r\n/* As far as we can tell, the lfence and sfence instructions are not */\r\n/* currently needed or useful for cached memory accesses. */\r\n\r\n/* Unfortunately mfence doesn't exist everywhere. */\r\n/* IsProcessorFeaturePresent(PF_COMPARE_EXCHANGE128) is */\r\n/* probably a conservative test for it? */\r\n\r\n#if defined(AO_USE_PENTIUM4_INSTRS)\r\n\r\nAO_INLINE void\r\nAO_nop_full(void)\r\n{\r\n __asm { mfence }\r\n}\r\n\r\n#define AO_HAVE_nop_full\r\n\r\n#else\r\n\r\n/* We could use the cpuid instruction. But that seems to be slower */\r\n/* than the default implementation based on test_and_set_full. Thus */\r\n/* we omit that bit of misinformation here. */\r\n\r\n#endif\r\n\r\nAO_INLINE AO_TS_VAL_t\r\nAO_test_and_set_full(volatile AO_TS_t *addr)\r\n{\r\n __asm\r\n {\r\n mov eax,0xff ; /* AO_TS_SET */\r\n mov ebx,addr ;\r\n xchg byte ptr [ebx],al ;\r\n }\r\n /* Ignore possible \"missing return value\" warning here. */\r\n}\r\n\r\n#define AO_HAVE_test_and_set_full\r\n\r\n#ifdef _WIN64\r\n# error wrong architecture\r\n#endif\r\n\r\n#ifdef AO_ASSUME_VISTA\r\n/* NEC LE-IT: whenever we run on a pentium class machine we have that\r\n * certain function */\r\n\r\n#include \"../standard_ao_double_t.h\"\r\n#pragma intrinsic (_InterlockedCompareExchange64)\r\n/* Returns nonzero if the comparison succeeded. */\r\nAO_INLINE int\r\nAO_compare_double_and_swap_double_full(volatile AO_double_t *addr,\r\n AO_t old_val1, AO_t old_val2,\r\n AO_t new_val1, AO_t new_val2)\r\n{\r\n __int64 oldv = (__int64)old_val1 | ((__int64)old_val2 << 32);\r\n __int64 newv = (__int64)new_val1 | ((__int64)new_val2 << 32);\r\n return _InterlockedCompareExchange64((__int64 volatile *)addr,\r\n newv, oldv) == oldv;\r\n}\r\n#define AO_HAVE_compare_double_and_swap_double_full\r\n\r\n#ifdef __cplusplus\r\nAO_INLINE int\r\nAO_double_compare_and_swap_full(volatile AO_double_t *addr,\r\n AO_double_t old_val, AO_double_t new_val)\r\n{\r\n return _InterlockedCompareExchange64((__int64 volatile *)addr,\r\n new_val.AO_whole, old_val.AO_whole) == old_val.AO_whole;\r\n}\r\n#define AO_HAVE_double_compare_and_swap_full\r\n#endif /* __cplusplus */\r\n#endif /* AO_ASSUME_VISTA */\r\n\r\n#include \"../ao_t_is_int.h\"\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/msftc/x86_64.h", "content": "/*\r\n * Copyright (c) 2003 Hewlett-Packard Development Company, L.P.\r\n *\r\n * Permission is hereby granted, free of charge, to any person obtaining a copy\r\n * of this software and associated documentation files (the \"Software\"), to deal\r\n * in the Software without restriction, including without limitation the rights\r\n * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\r\n * copies of the Software, and to permit persons to whom the Software is\r\n * furnished to do so, subject to the following conditions:\r\n *\r\n * The above copyright notice and this permission notice shall be included in\r\n * all copies or substantial portions of the Software.\r\n *\r\n * THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\r\n * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\r\n * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\r\n * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\r\n * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\r\n * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\r\n * SOFTWARE.\r\n */\r\n\r\n#include \"../all_aligned_atomic_load_store.h\"\r\n\r\n/* Real X86 implementations appear */\r\n/* to enforce ordering between memory operations, EXCEPT that a later */\r\n/* read can pass earlier writes, presumably due to the visible */\r\n/* presence of store buffers. */\r\n/* We ignore the fact that the official specs */\r\n/* seem to be much weaker (and arguably too weak to be usable). */\r\n\r\n#include \"../ordered_except_wr.h\"\r\n\r\n#ifdef AO_ASM_X64_AVAILABLE\r\n# include \"../test_and_set_t_is_char.h\"\r\n#else\r\n# include \"../test_and_set_t_is_ao_t.h\"\r\n#endif\r\n\r\n#include \"../standard_ao_double_t.h\"\r\n\r\n#include \r\n /* Seems like over-kill, but that's what MSDN recommends. */\r\n /* And apparently winbase.h is not always self-contained. */\r\n\r\n/* Assume _MSC_VER >= 1400 */\r\n#include \r\n\r\n#pragma intrinsic (_ReadWriteBarrier)\r\n\r\n#pragma intrinsic (_InterlockedIncrement64)\r\n#pragma intrinsic (_InterlockedDecrement64)\r\n#pragma intrinsic (_InterlockedExchange64)\r\n#pragma intrinsic (_InterlockedExchangeAdd64)\r\n#pragma intrinsic (_InterlockedCompareExchange64)\r\n\r\nAO_INLINE AO_t\r\nAO_fetch_and_add_full (volatile AO_t *p, AO_t incr)\r\n{\r\n return _InterlockedExchangeAdd64((LONGLONG volatile *)p, (LONGLONG)incr);\r\n}\r\n\r\n#define AO_HAVE_fetch_and_add_full\r\n\r\nAO_INLINE AO_t\r\nAO_fetch_and_add1_full (volatile AO_t *p)\r\n{\r\n return _InterlockedIncrement64((LONGLONG volatile *)p) - 1;\r\n}\r\n\r\n#define AO_HAVE_fetch_and_add1_full\r\n\r\nAO_INLINE AO_t\r\nAO_fetch_and_sub1_full (volatile AO_t *p)\r\n{\r\n return _InterlockedDecrement64((LONGLONG volatile *)p) + 1;\r\n}\r\n\r\n#define AO_HAVE_fetch_and_sub1_full\r\n\r\nAO_INLINE int\r\nAO_compare_and_swap_full(volatile AO_t *addr,\r\n AO_t old, AO_t new_val)\r\n{\r\n return _InterlockedCompareExchange64((LONGLONG volatile *)addr,\r\n (LONGLONG)new_val, (LONGLONG)old)\r\n == (LONGLONG)old;\r\n}\r\n\r\n#define AO_HAVE_compare_and_swap_full\r\n\r\n/* As far as we can tell, the lfence and sfence instructions are not */\r\n/* currently needed or useful for cached memory accesses. */\r\n\r\n#ifdef AO_ASM_X64_AVAILABLE\r\n\r\nAO_INLINE void\r\nAO_nop_full(void)\r\n{\r\n /* Note: \"mfence\" (SSE2) is supported on all x86_64/amd64 chips. */\r\n __asm { mfence }\r\n}\r\n\r\n#define AO_HAVE_nop_full\r\n\r\nAO_INLINE AO_TS_VAL_t\r\nAO_test_and_set_full(volatile AO_TS_t *addr)\r\n{\r\n __asm\r\n {\r\n mov rax,AO_TS_SET ;\r\n mov rbx,addr ;\r\n xchg byte ptr [rbx],al ;\r\n }\r\n}\r\n\r\n#define AO_HAVE_test_and_set_full\r\n\r\n#endif /* AO_ASM_X64_AVAILABLE */\r\n\r\n#ifdef AO_CMPXCHG16B_AVAILABLE\r\n\r\n/* AO_compare_double_and_swap_double_full needs implementation for Win64.\r\n * Also see ../gcc/x86_64.h for partial old Opteron workaround.\r\n */\r\n\r\n# if _MSC_VER >= 1500\r\n\r\n#pragma intrinsic (_InterlockedCompareExchange128)\r\n\r\nAO_INLINE int\r\nAO_compare_double_and_swap_double_full(volatile AO_double_t *addr,\r\n AO_t old_val1, AO_t old_val2,\r\n AO_t new_val1, AO_t new_val2)\r\n{\r\n __int64 comparandResult[2];\r\n comparandResult[0] = old_val1; /* low */\r\n comparandResult[1] = old_val2; /* high */\r\n return _InterlockedCompareExchange128((volatile __int64 *)addr,\r\n new_val2 /* high */, new_val1 /* low */, comparandResult);\r\n}\r\n\r\n# define AO_HAVE_compare_double_and_swap_double_full\r\n\r\n# elif defined(AO_ASM_X64_AVAILABLE)\r\n\r\n /* If there is no intrinsic _InterlockedCompareExchange128 then we\r\n * need basically what's given below.\r\n */\r\n\r\nAO_INLINE int\r\nAO_compare_double_and_swap_double_full(volatile AO_double_t *addr,\r\n AO_t old_val1, AO_t old_val2,\r\n AO_t new_val1, AO_t new_val2)\r\n{\r\n __asm\r\n {\r\n mov rdx,QWORD PTR [old_val2] ;\r\n mov rax,QWORD PTR [old_val1] ;\r\n mov rcx,QWORD PTR [new_val2] ;\r\n mov rbx,QWORD PTR [new_val1] ;\r\n lock cmpxchg16b [addr] ;\r\n setz rax ;\r\n }\r\n}\r\n\r\n# define AO_HAVE_compare_double_and_swap_double_full\r\n\r\n# endif /* _MSC_VER >= 1500 || AO_ASM_X64_AVAILABLE */\r\n\r\n#endif /* AO_CMPXCHG16B_AVAILABLE */\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/ordered.h", "content": "/*\r\n * Copyright (c) 2003 Hewlett-Packard Development Company, L.P.\r\n *\r\n * Permission is hereby granted, free of charge, to any person obtaining a copy\r\n * of this software and associated documentation files (the \"Software\"), to deal\r\n * in the Software without restriction, including without limitation the rights\r\n * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\r\n * copies of the Software, and to permit persons to whom the Software is\r\n * furnished to do so, subject to the following conditions:\r\n *\r\n * The above copyright notice and this permission notice shall be included in\r\n * all copies or substantial portions of the Software.\r\n *\r\n * THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\r\n * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\r\n * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\r\n * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\r\n * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\r\n * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\r\n * SOFTWARE.\r\n */\r\n\r\n/*\r\n * These are common definitions for architectures that provide processor\r\n * ordered memory operations.\r\n */\r\n\r\n#include \"ordered_except_wr.h\"\r\n\r\nAO_INLINE void\r\nAO_nop_full(void)\r\n{\r\n AO_compiler_barrier();\r\n}\r\n\r\n#define AO_HAVE_nop_full\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/ordered_except_wr.h", "content": "/*\r\n * Copyright (c) 2003 by Hewlett-Packard Company. All rights reserved.\r\n *\r\n * Permission is hereby granted, free of charge, to any person obtaining a copy\r\n * of this software and associated documentation files (the \"Software\"), to deal\r\n * in the Software without restriction, including without limitation the rights\r\n * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\r\n * copies of the Software, and to permit persons to whom the Software is\r\n * furnished to do so, subject to the following conditions:\r\n *\r\n * The above copyright notice and this permission notice shall be included in\r\n * all copies or substantial portions of the Software.\r\n *\r\n * THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\r\n * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\r\n * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\r\n * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\r\n * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\r\n * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\r\n * SOFTWARE.\r\n */\r\n\r\n/*\r\n * These are common definitions for architectures that provide processor\r\n * ordered memory operations except that a later read may pass an\r\n * earlier write. Real x86 implementations seem to be in this category,\r\n * except apparently for some IDT WinChips, which we ignore.\r\n */\r\n\r\n#include \"read_ordered.h\"\r\n\r\nAO_INLINE void\r\nAO_nop_write(void)\r\n{\r\n AO_compiler_barrier();\r\n /* sfence according to Intel docs. Pentium 3 and up. */\r\n /* Unnecessary for cached accesses? */\r\n}\r\n\r\n#define AO_HAVE_NOP_WRITE\r\n\r\n#if defined(AO_HAVE_store)\r\n\r\nAO_INLINE void\r\nAO_store_write(volatile AO_t *addr, AO_t val)\r\n{\r\n AO_compiler_barrier();\r\n AO_store(addr, val);\r\n}\r\n# define AO_HAVE_store_write\r\n\r\n# define AO_store_release(addr, val) AO_store_write(addr, val)\r\n# define AO_HAVE_store_release\r\n\r\n#endif /* AO_HAVE_store */\r\n\r\n#if defined(AO_HAVE_char_store)\r\n\r\nAO_INLINE void\r\nAO_char_store_write(volatile unsigned char *addr, unsigned char val)\r\n{\r\n AO_compiler_barrier();\r\n AO_char_store(addr, val);\r\n}\r\n# define AO_HAVE_char_store_write\r\n\r\n# define AO_char_store_release(addr, val) AO_char_store_write(addr, val)\r\n# define AO_HAVE_char_store_release\r\n\r\n#endif /* AO_HAVE_char_store */\r\n\r\n#if defined(AO_HAVE_short_store)\r\n\r\nAO_INLINE void\r\nAO_short_store_write(volatile unsigned short *addr, unsigned short val)\r\n{\r\n AO_compiler_barrier();\r\n AO_short_store(addr, val);\r\n}\r\n# define AO_HAVE_short_store_write\r\n\r\n# define AO_short_store_release(addr, val) AO_short_store_write(addr, val)\r\n# define AO_HAVE_short_store_release\r\n\r\n#endif /* AO_HAVE_short_store */\r\n\r\n#if defined(AO_HAVE_int_store)\r\n\r\nAO_INLINE void\r\nAO_int_store_write(volatile unsigned int *addr, unsigned int val)\r\n{\r\n AO_compiler_barrier();\r\n AO_int_store(addr, val);\r\n}\r\n# define AO_HAVE_int_store_write\r\n\r\n# define AO_int_store_release(addr, val) AO_int_store_write(addr, val)\r\n# define AO_HAVE_int_store_release\r\n\r\n#endif /* AO_HAVE_int_store */\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/read_ordered.h", "content": "/*\r\n * Copyright (c) 2003 by Hewlett-Packard Company. All rights reserved.\r\n *\r\n * Permission is hereby granted, free of charge, to any person obtaining a copy\r\n * of this software and associated documentation files (the \"Software\"), to deal\r\n * in the Software without restriction, including without limitation the rights\r\n * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\r\n * copies of the Software, and to permit persons to whom the Software is\r\n * furnished to do so, subject to the following conditions:\r\n *\r\n * The above copyright notice and this permission notice shall be included in\r\n * all copies or substantial portions of the Software.\r\n *\r\n * THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\r\n * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\r\n * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\r\n * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\r\n * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\r\n * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\r\n * SOFTWARE.\r\n */\r\n\r\n/*\r\n * These are common definitions for architectures that provide processor\r\n * ordered memory operations except that a later read may pass an\r\n * earlier write. Real x86 implementations seem to be in this category,\r\n * except apparently for some IDT WinChips, which we ignore.\r\n */\r\n\r\nAO_INLINE void\r\nAO_nop_read(void)\r\n{\r\n AO_compiler_barrier();\r\n}\r\n\r\n#define AO_HAVE_NOP_READ\r\n\r\n#ifdef AO_HAVE_load\r\n\r\nAO_INLINE AO_t\r\nAO_load_read(const volatile AO_t *addr)\r\n{\r\n AO_t result = AO_load(addr);\r\n AO_compiler_barrier();\r\n return result;\r\n}\r\n#define AO_HAVE_load_read\r\n\r\n#define AO_load_acquire(addr) AO_load_read(addr)\r\n#define AO_HAVE_load_acquire\r\n\r\n#endif /* AO_HAVE_load */\r\n\r\n#ifdef AO_HAVE_char_load\r\n\r\nAO_INLINE AO_t\r\nAO_char_load_read(const volatile unsigned char *addr)\r\n{\r\n AO_t result = AO_char_load(addr);\r\n AO_compiler_barrier();\r\n return result;\r\n}\r\n#define AO_HAVE_char_load_read\r\n\r\n#define AO_char_load_acquire(addr) AO_char_load_read(addr)\r\n#define AO_HAVE_char_load_acquire\r\n\r\n#endif /* AO_HAVE_char_load */\r\n\r\n#ifdef AO_HAVE_short_load\r\n\r\nAO_INLINE AO_t\r\nAO_short_load_read(const volatile unsigned short *addr)\r\n{\r\n AO_t result = AO_short_load(addr);\r\n AO_compiler_barrier();\r\n return result;\r\n}\r\n#define AO_HAVE_short_load_read\r\n\r\n#define AO_short_load_acquire(addr) AO_short_load_read(addr)\r\n#define AO_HAVE_short_load_acquire\r\n\r\n#endif /* AO_HAVE_short_load */\r\n\r\n#ifdef AO_HAVE_int_load\r\n\r\nAO_INLINE AO_t\r\nAO_int_load_read(const volatile unsigned int *addr)\r\n{\r\n AO_t result = AO_int_load(addr);\r\n AO_compiler_barrier();\r\n return result;\r\n}\r\n#define AO_HAVE_int_load_read\r\n\r\n#define AO_int_load_acquire(addr) AO_int_load_read(addr)\r\n#define AO_HAVE_int_load_acquire\r\n\r\n#endif /* AO_HAVE_int_load */\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/short_acquire_release_volatile.h", "content": "/*\r\n * Copyright (c) 2003-2004 Hewlett-Packard Development Company, L.P.\r\n *\r\n * Permission is hereby granted, free of charge, to any person obtaining a copy\r\n * of this software and associated documentation files (the \"Software\"), to deal\r\n * in the Software without restriction, including without limitation the rights\r\n * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\r\n * copies of the Software, and to permit persons to whom the Software is\r\n * furnished to do so, subject to the following conditions:\r\n *\r\n * The above copyright notice and this permission notice shall be included in\r\n * all copies or substantial portions of the Software.\r\n *\r\n * THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\r\n * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\r\n * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\r\n * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\r\n * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\r\n * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\r\n * SOFTWARE.\r\n */\r\n\r\n/*\r\n * This file adds definitions appropriate for environments in which an unsigned short\r\n * volatile load has acquire semantics, and an unsigned short volatile store has release\r\n * semantics. This is true with the standard Itanium ABI.\r\n */\r\n#if !defined(AO_GCC_BARRIER)\r\n# if defined(__GNUC__)\r\n# define AO_GCC_BARRIER() AO_compiler_barrier()\r\n# else\r\n# define AO_GCC_BARRIER()\r\n# endif\r\n#endif\r\n\r\nAO_INLINE unsigned short\r\nAO_short_load_acquire(const volatile unsigned short *p)\r\n{\r\n unsigned short result = *p;\r\n /* A normal volatile load generates an ld.acq */\r\n AO_GCC_BARRIER();\r\n return result;\r\n}\r\n#define AO_HAVE_short_load_acquire\r\n\r\nAO_INLINE void\r\nAO_short_store_release(volatile unsigned short *p, unsigned short val)\r\n{\r\n AO_GCC_BARRIER();\r\n /* A normal volatile store generates an st.rel */\r\n *p = val;\r\n}\r\n#define AO_HAVE_short_store_release\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/short_aligned_atomic_load_store.h", "content": "/*\r\n * Copyright (c) 2004 Hewlett-Packard Development Company, L.P.\r\n *\r\n * Permission is hereby granted, free of charge, to any person obtaining a copy\r\n * of this software and associated documentation files (the \"Software\"), to deal\r\n * in the Software without restriction, including without limitation the rights\r\n * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\r\n * copies of the Software, and to permit persons to whom the Software is\r\n * furnished to do so, subject to the following conditions:\r\n *\r\n * The above copyright notice and this permission notice shall be included in\r\n * all copies or substantial portions of the Software.\r\n *\r\n * THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\r\n * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\r\n * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\r\n * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\r\n * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\r\n * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\r\n * SOFTWARE.\r\n */\r\n\r\n/*\r\n * Definitions for architectures on which loads and stores of unsigned short\r\n * are atomic for all legal alignments.\r\n */\r\n\r\nAO_INLINE unsigned short\r\nAO_short_load(const volatile unsigned short *addr)\r\n{\r\n assert(((size_t)addr & (sizeof(unsigned short) - 1)) == 0);\r\n /* Cast away the volatile for architectures like IA64 where */\r\n /* volatile adds barrier semantics. */\r\n return (*(unsigned short *)addr);\r\n}\r\n\r\n#define AO_HAVE_short_load\r\n\r\nAO_INLINE void\r\nAO_short_store(volatile unsigned short *addr, unsigned short new_val)\r\n{\r\n assert(((size_t)addr & (sizeof(unsigned short) - 1)) == 0);\r\n (*(unsigned short *)addr) = new_val;\r\n}\r\n\r\n#define AO_HAVE_short_store\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/short_atomic_load_store.h", "content": "/*\r\n * Copyright (c) 2003 by Hewlett-Packard Company. All rights reserved.\r\n *\r\n * Permission is hereby granted, free of charge, to any person obtaining a copy\r\n * of this software and associated documentation files (the \"Software\"), to deal\r\n * in the Software without restriction, including without limitation the rights\r\n * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\r\n * copies of the Software, and to permit persons to whom the Software is\r\n * furnished to do so, subject to the following conditions:\r\n *\r\n * The above copyright notice and this permission notice shall be included in\r\n * all copies or substantial portions of the Software.\r\n *\r\n * THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\r\n * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\r\n * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\r\n * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\r\n * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\r\n * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\r\n * SOFTWARE.\r\n */\r\n\r\n/*\r\n * Definitions for architectures on which loads and stores of unsigned short\r\n * are atomic for all legal alignments.\r\n */\r\n\r\nAO_INLINE unsigned short\r\nAO_short_load(const volatile unsigned short *addr)\r\n{\r\n /* Cast away the volatile for architectures like IA64 where */\r\n /* volatile adds barrier semantics. */\r\n return (*(const unsigned short *)addr);\r\n}\r\n\r\n#define AO_HAVE_short_load\r\n\r\nAO_INLINE void\r\nAO_short_store(volatile unsigned short *addr, unsigned short new_val)\r\n{\r\n (*(unsigned short *)addr) = new_val;\r\n}\r\n\r\n#define AO_HAVE_short_store\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/standard_ao_double_t.h", "content": "/* NEC LE-IT: For 64Bit OS we extend the double type to hold two int64's\r\n*\r\n* x86-64: __m128 serves as placeholder which also requires the compiler\r\n* to align it on 16 byte boundary (as required by cmpxchg16.\r\n* Similar things could be done for PowerPC 64bit using a VMX data type... */\r\n\r\n#if (defined(__x86_64__) && defined(__GNUC__)) || defined(_WIN64)\r\n# include \r\n typedef __m128 double_ptr_storage;\r\n#elif defined(_WIN32) && !defined(__GNUC__)\r\n typedef unsigned __int64 double_ptr_storage;\r\n#else\r\n typedef unsigned long long double_ptr_storage;\r\n#endif\r\n\r\n# define AO_HAVE_DOUBLE_PTR_STORAGE\r\n\r\ntypedef union {\r\n double_ptr_storage AO_whole;\r\n struct {AO_t AO_v1; AO_t AO_v2;} AO_parts;\r\n} AO_double_t;\r\n\r\n#define AO_HAVE_double_t\r\n#define AO_val1 AO_parts.AO_v1\r\n#define AO_val2 AO_parts.AO_v2\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/sunc/sparc.h", "content": "/*\r\n * Copyright (c) 2004 Hewlett-Packard Development Company, L.P.\r\n * \r\n * Permission is hereby granted, free of charge, to any person obtaining a copy\r\n * of this software and associated documentation files (the \"Software\"), to deal\r\n * in the Software without restriction, including without limitation the rights\r\n * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\r\n * copies of the Software, and to permit persons to whom the Software is\r\n * furnished to do so, subject to the following conditions:\r\n * \r\n * The above copyright notice and this permission notice shall be included in\r\n * all copies or substantial portions of the Software.\r\n * \r\n * THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\r\n * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\r\n * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\r\n * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\r\n * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\r\n * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\r\n * SOFTWARE. \r\n */\r\n\r\n#include \"../all_atomic_load_store.h\"\r\n\r\n/* Real SPARC code uses TSO: */\r\n#include \"../ordered_except_wr.h\"\r\n\r\n/* Test_and_set location is just a byte.\t\t*/\r\n#include \"../test_and_set_t_is_char.h\"\r\n\r\nextern AO_TS_VAL_t\r\nAO_test_and_set_full(volatile AO_TS_t *addr);\r\n/* Implemented in separate .S file, for now.\t*/\r\n\r\n#define AO_HAVE_test_and_set_full\r\n\r\n/* FIXME: Like the gcc version, this needs to be extended for V8 \t*/\r\n/* and V9.\t\t\t\t\t\t\t\t*/\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/sunc/x86.h", "content": "/*\r\n * Copyright (c) 1991-1994 by Xerox Corporation. All rights reserved.\r\n * Copyright (c) 1996-1999 by Silicon Graphics. All rights reserved.\r\n * Copyright (c) 1999-2003 by Hewlett-Packard Company. All rights reserved.\r\n *\r\n *\r\n * THIS MATERIAL IS PROVIDED AS IS, WITH ABSOLUTELY NO WARRANTY EXPRESSED\r\n * OR IMPLIED. ANY USE IS AT YOUR OWN RISK.\r\n *\r\n * Permission is hereby granted to use or copy this program\r\n * for any purpose, provided the above notices are retained on all copies.\r\n * Permission to modify the code and to distribute modified code is granted,\r\n * provided the above notices are retained, and a notice that the code was\r\n * modified is included with the above copyright notice.\r\n *\r\n * Some of the machine specific code was borrowed from our GC distribution.\r\n */\r\n\r\n/* The following really assume we have a 486 or better. */\r\n\r\n#include \"../all_aligned_atomic_load_store.h\"\r\n\r\n/* Real X86 implementations, except for some old WinChips, appear */\r\n/* to enforce ordering between memory operations, EXCEPT that a later */\r\n/* read can pass earlier writes, presumably due to the visible */\r\n/* presence of store buffers. */\r\n/* We ignore both the WinChips, and the fact that the official specs */\r\n/* seem to be much weaker (and arguably too weak to be usable). */\r\n\r\n#include \"../ordered_except_wr.h\"\r\n\r\n#include \"../test_and_set_t_is_char.h\"\r\n\r\n#include \"../standard_ao_double_t.h\"\r\n\r\n#if defined(AO_USE_PENTIUM4_INSTRS)\r\nAO_INLINE void\r\nAO_nop_full(void)\r\n{\r\n __asm__ __volatile__(\"mfence\" : : : \"memory\");\r\n}\r\n\r\n#define AO_HAVE_nop_full\r\n\r\n#else\r\n\r\n/* We could use the cpuid instruction. But that seems to be slower */\r\n/* than the default implementation based on test_and_set_full. Thus */\r\n/* we omit that bit of misinformation here. */\r\n\r\n#endif\r\n\r\n/* As far as we can tell, the lfence and sfence instructions are not */\r\n/* currently needed or useful for cached memory accesses. */\r\n\r\n/* Really only works for 486 and later */\r\nAO_INLINE AO_t\r\nAO_fetch_and_add_full (volatile AO_t *p, AO_t incr)\r\n{\r\n AO_t result;\r\n\r\n __asm__ __volatile__ (\"lock; xaddl %0, %1\" :\r\n \"=r\" (result), \"=m\" (*p) : \"0\" (incr) /* , \"m\" (*p) */\r\n : \"memory\");\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_fetch_and_add_full\r\n\r\nAO_INLINE unsigned char\r\nAO_char_fetch_and_add_full (volatile unsigned char *p, unsigned char incr)\r\n{\r\n unsigned char result;\r\n\r\n __asm__ __volatile__ (\"lock; xaddb %0, %1\" :\r\n \"=q\" (result), \"=m\" (*p) : \"0\" (incr) /* , \"m\" (*p) */\r\n : \"memory\");\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_char_fetch_and_add_full\r\n\r\nAO_INLINE unsigned short\r\nAO_short_fetch_and_add_full (volatile unsigned short *p, unsigned short incr)\r\n{\r\n unsigned short result;\r\n\r\n __asm__ __volatile__ (\"lock; xaddw %0, %1\" :\r\n \"=r\" (result), \"=m\" (*p) : \"0\" (incr) /* , \"m\" (*p) */\r\n : \"memory\");\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_short_fetch_and_add_full\r\n\r\n/* Really only works for 486 and later */\r\nAO_INLINE void\r\nAO_or_full (volatile AO_t *p, AO_t incr)\r\n{\r\n __asm__ __volatile__ (\"lock; orl %1, %0\" :\r\n \"=m\" (*p) : \"r\" (incr) /* , \"m\" (*p) */\r\n : \"memory\");\r\n}\r\n\r\n#define AO_HAVE_or_full\r\n\r\nAO_INLINE AO_TS_VAL_t\r\nAO_test_and_set_full(volatile AO_TS_t *addr)\r\n{\r\n AO_TS_t oldval;\r\n /* Note: the \"xchg\" instruction does not need a \"lock\" prefix */\r\n __asm__ __volatile__(\"xchg %0, %1\"\r\n : \"=q\"(oldval), \"=m\"(*addr)\r\n : \"0\"(0xff) /* , \"m\"(*addr) */\r\n : \"memory\");\r\n return (AO_TS_VAL_t)oldval;\r\n}\r\n\r\n#define AO_HAVE_test_and_set_full\r\n\r\n/* Returns nonzero if the comparison succeeded. */\r\nAO_INLINE int\r\nAO_compare_and_swap_full(volatile AO_t *addr,\r\n AO_t old, AO_t new_val)\r\n{\r\n char result;\r\n __asm__ __volatile__(\"lock; cmpxchgl %2, %0; setz %1\"\r\n : \"=m\"(*addr), \"=a\"(result)\r\n : \"r\" (new_val), \"a\"(old) : \"memory\");\r\n return (int) result;\r\n}\r\n\r\n#define AO_HAVE_compare_and_swap_full\r\n\r\n#if 0\r\n/* FIXME: not tested (and probably wrong). Besides, */\r\n/* it tickles a bug in Sun C 5.10 (when optimizing). */\r\n/* Returns nonzero if the comparison succeeded. */\r\n/* Really requires at least a Pentium. */\r\nAO_INLINE int\r\nAO_compare_double_and_swap_double_full(volatile AO_double_t *addr,\r\n AO_t old_val1, AO_t old_val2,\r\n AO_t new_val1, AO_t new_val2)\r\n{\r\n char result;\r\n#if __PIC__\r\n /* If PIC is turned on, we can't use %ebx as it is reserved for the\r\n GOT pointer. We can save and restore %ebx because GCC won't be\r\n using it for anything else (such as any of the m operands) */\r\n __asm__ __volatile__(\"pushl %%ebx;\" /* save ebx used for PIC GOT ptr */\r\n \"movl %6,%%ebx;\" /* move new_val2 to %ebx */\r\n \"lock; cmpxchg8b %0; setz %1;\"\r\n \"pop %%ebx;\" /* restore %ebx */\r\n : \"=m\"(*addr), \"=a\"(result)\r\n : \"m\"(*addr), \"d\" (old_val2), \"a\" (old_val1),\r\n \"c\" (new_val2), \"m\" (new_val1) : \"memory\");\r\n#else\r\n /* We can't just do the same thing in non-PIC mode, because GCC\r\n * might be using %ebx as the memory operand. We could have ifdef'd\r\n * in a clobber, but there's no point doing the push/pop if we don't\r\n * have to. */\r\n __asm__ __volatile__(\"lock; cmpxchg8b %0; setz %1;\"\r\n : \"=m\"(*addr), \"=a\"(result)\r\n : /* \"m\"(*addr), */ \"d\" (old_val2), \"a\" (old_val1),\r\n \"c\" (new_val2), \"b\" (new_val1) : \"memory\");\r\n#endif\r\n return (int) result;\r\n}\r\n\r\n#define AO_HAVE_compare_double_and_swap_double_full\r\n#endif\r\n\r\n#include \"../ao_t_is_int.h\"\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/sunc/x86_64.h", "content": "/*\r\n * Copyright (c) 1991-1994 by Xerox Corporation. All rights reserved.\r\n * Copyright (c) 1996-1999 by Silicon Graphics. All rights reserved.\r\n * Copyright (c) 1999-2003 by Hewlett-Packard Company. All rights reserved.\r\n *\r\n *\r\n * THIS MATERIAL IS PROVIDED AS IS, WITH ABSOLUTELY NO WARRANTY EXPRESSED\r\n * OR IMPLIED. ANY USE IS AT YOUR OWN RISK.\r\n *\r\n * Permission is hereby granted to use or copy this program\r\n * for any purpose, provided the above notices are retained on all copies.\r\n * Permission to modify the code and to distribute modified code is granted,\r\n * provided the above notices are retained, and a notice that the code was\r\n * modified is included with the above copyright notice.\r\n *\r\n * Some of the machine specific code was borrowed from our GC distribution.\r\n */\r\n\r\n#include \"../all_aligned_atomic_load_store.h\"\r\n\r\n/* Real X86 implementations, appear */\r\n/* to enforce ordering between memory operations, EXCEPT that a later */\r\n/* read can pass earlier writes, presumably due to the visible */\r\n/* presence of store buffers. */\r\n/* We ignore the fact that the official specs */\r\n/* seem to be much weaker (and arguably too weak to be usable). */\r\n\r\n#include \"../ordered_except_wr.h\"\r\n\r\n#include \"../test_and_set_t_is_char.h\"\r\n\r\n#include \"../standard_ao_double_t.h\"\r\n\r\nAO_INLINE void\r\nAO_nop_full(void)\r\n{\r\n /* Note: \"mfence\" (SSE2) is supported on all x86_64/amd64 chips. */\r\n __asm__ __volatile__(\"mfence\" : : : \"memory\");\r\n}\r\n\r\n#define AO_HAVE_nop_full\r\n\r\n/* As far as we can tell, the lfence and sfence instructions are not */\r\n/* currently needed or useful for cached memory accesses. */\r\n\r\nAO_INLINE AO_t\r\nAO_fetch_and_add_full (volatile AO_t *p, AO_t incr)\r\n{\r\n AO_t result;\r\n\r\n __asm__ __volatile__ (\"lock; xaddq %0, %1\" :\r\n \"=r\" (result), \"=m\" (*p) : \"0\" (incr) /* , \"m\" (*p) */\r\n : \"memory\");\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_fetch_and_add_full\r\n\r\nAO_INLINE unsigned char\r\nAO_char_fetch_and_add_full (volatile unsigned char *p, unsigned char incr)\r\n{\r\n unsigned char result;\r\n\r\n __asm__ __volatile__ (\"lock; xaddb %0, %1\" :\r\n \"=q\" (result), \"=m\" (*p) : \"0\" (incr) /* , \"m\" (*p) */\r\n : \"memory\");\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_char_fetch_and_add_full\r\n\r\nAO_INLINE unsigned short\r\nAO_short_fetch_and_add_full (volatile unsigned short *p, unsigned short incr)\r\n{\r\n unsigned short result;\r\n\r\n __asm__ __volatile__ (\"lock; xaddw %0, %1\" :\r\n \"=r\" (result), \"=m\" (*p) : \"0\" (incr) /* , \"m\" (*p) */\r\n : \"memory\");\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_short_fetch_and_add_full\r\n\r\nAO_INLINE unsigned int\r\nAO_int_fetch_and_add_full (volatile unsigned int *p, unsigned int incr)\r\n{\r\n unsigned int result;\r\n\r\n __asm__ __volatile__ (\"lock; xaddl %0, %1\" :\r\n \"=r\" (result), \"=m\" (*p) : \"0\" (incr) /* , \"m\" (*p) */\r\n : \"memory\");\r\n return result;\r\n}\r\n\r\n#define AO_HAVE_int_fetch_and_add_full\r\n\r\nAO_INLINE void\r\nAO_or_full (volatile AO_t *p, AO_t incr)\r\n{\r\n __asm__ __volatile__ (\"lock; orq %1, %0\" :\r\n \"=m\" (*p) : \"r\" (incr) /* , \"m\" (*p) */\r\n : \"memory\");\r\n}\r\n\r\n#define AO_HAVE_or_full\r\n\r\nAO_INLINE AO_TS_VAL_t\r\nAO_test_and_set_full(volatile AO_TS_t *addr)\r\n{\r\n AO_TS_t oldval;\r\n /* Note: the \"xchg\" instruction does not need a \"lock\" prefix */\r\n __asm__ __volatile__(\"xchg %0, %1\"\r\n : \"=q\"(oldval), \"=m\"(*addr)\r\n : \"0\"(0xff) /* , \"m\"(*addr) */\r\n : \"memory\");\r\n return (AO_TS_VAL_t)oldval;\r\n}\r\n\r\n#define AO_HAVE_test_and_set_full\r\n\r\n/* Returns nonzero if the comparison succeeded. */\r\nAO_INLINE int\r\nAO_compare_and_swap_full(volatile AO_t *addr,\r\n AO_t old, AO_t new_val)\r\n{\r\n char result;\r\n __asm__ __volatile__(\"lock; cmpxchgq %2, %0; setz %1\"\r\n : \"=m\"(*addr), \"=a\"(result)\r\n : \"r\" (new_val), \"a\"(old) : \"memory\");\r\n return (int) result;\r\n}\r\n\r\n#define AO_HAVE_compare_and_swap_full\r\n\r\n#ifdef AO_CMPXCHG16B_AVAILABLE\r\n/* NEC LE-IT: older AMD Opterons are missing this instruction.\r\n * On these machines SIGILL will be thrown.\r\n * Define AO_WEAK_DOUBLE_CAS_EMULATION to have an emulated\r\n * (lock based) version available */\r\n/* HB: Changed this to not define either by default. There are\r\n * enough machines and tool chains around on which cmpxchg16b\r\n * doesn't work. And the emulation is unsafe by our usual rules.\r\n * Hoewever both are clearly useful in certain cases.\r\n */\r\nAO_INLINE int\r\nAO_compare_double_and_swap_double_full(volatile AO_double_t *addr,\r\n AO_t old_val1, AO_t old_val2,\r\n AO_t new_val1, AO_t new_val2)\r\n{\r\n char result;\r\n __asm__ __volatile__(\"lock; cmpxchg16b %0; setz %1\"\r\n : \"=m\"(*addr), \"=a\"(result)\r\n : \"m\"(*addr), \"d\" (old_val2), \"a\" (old_val1),\r\n \"c\" (new_val2), \"b\" (new_val1) : \"memory\");\r\n return (int) result;\r\n}\r\n#define AO_HAVE_compare_double_and_swap_double_full\r\n#else\r\n/* this one provides spinlock based emulation of CAS implemented in */\r\n/* atomic_ops.c. We probably do not want to do this here, since it is */\r\n/* not atomic with respect to other kinds of updates of *addr. On the */\r\n/* other hand, this may be a useful facility on occasion. */\r\n#ifdef AO_WEAK_DOUBLE_CAS_EMULATION\r\nint AO_compare_double_and_swap_double_emulation(volatile AO_double_t *addr,\r\n AO_t old_val1, AO_t old_val2,\r\n AO_t new_val1, AO_t new_val2);\r\n\r\nAO_INLINE int\r\nAO_compare_double_and_swap_double_full(volatile AO_double_t *addr,\r\n AO_t old_val1, AO_t old_val2,\r\n AO_t new_val1, AO_t new_val2)\r\n{\r\n return AO_compare_double_and_swap_double_emulation(addr,\r\n old_val1, old_val2,\r\n new_val1, new_val2);\r\n}\r\n#define AO_HAVE_compare_double_and_swap_double_full\r\n#endif /* AO_WEAK_DOUBLE_CAS_EMULATION */\r\n#endif /* AO_CMPXCHG16B_AVAILABLE */\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/test_and_set_t_is_ao_t.h", "content": "/*\r\n * Copyright (c) 2004 Hewlett-Packard Development Company, L.P.\r\n *\r\n * Permission is hereby granted, free of charge, to any person obtaining a copy\r\n * of this software and associated documentation files (the \"Software\"), to deal\r\n * in the Software without restriction, including without limitation the rights\r\n * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\r\n * copies of the Software, and to permit persons to whom the Software is\r\n * furnished to do so, subject to the following conditions:\r\n *\r\n * The above copyright notice and this permission notice shall be included in\r\n * all copies or substantial portions of the Software.\r\n *\r\n * THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\r\n * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\r\n * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\r\n * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\r\n * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\r\n * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\r\n * SOFTWARE.\r\n */\r\n\r\n/*\r\n * These are common definitions for architectures on which test_and_set\r\n * operates on pointer-sized quantities, the \"clear\" value contains\r\n * all zeroes, and the \"set\" value contains only one lowest bit set.\r\n * This can be used if test_and_set is synthesized from compare_and_swap.\r\n */\r\ntypedef enum {AO_TS_clear = 0, AO_TS_set = 1} AO_TS_val;\r\n#define AO_TS_VAL_t AO_TS_val\r\n#define AO_TS_CLEAR AO_TS_clear\r\n#define AO_TS_SET AO_TS_set\r\n\r\n#define AO_TS_t AO_t\r\n\r\n#define AO_AO_TS_T 1\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops/sysdeps/test_and_set_t_is_char.h", "content": "/*\r\n * Copyright (c) 2004 Hewlett-Packard Development Company, L.P.\r\n * \r\n * Permission is hereby granted, free of charge, to any person obtaining a copy\r\n * of this software and associated documentation files (the \"Software\"), to deal\r\n * in the Software without restriction, including without limitation the rights\r\n * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\r\n * copies of the Software, and to permit persons to whom the Software is\r\n * furnished to do so, subject to the following conditions:\r\n * \r\n * The above copyright notice and this permission notice shall be included in\r\n * all copies or substantial portions of the Software.\r\n * \r\n * THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\r\n * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\r\n * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\r\n * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\r\n * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\r\n * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\r\n * SOFTWARE. \r\n */ \r\n\r\n/*\r\n * These are common definitions for architectures on which test_and_set\r\n * operates on byte sized quantities, the \"clear\" value contains\r\n * all zeroes, and the \"set\" value contains all ones.\r\n */\r\n\r\n#define AO_TS_t unsigned char\r\ntypedef enum {AO_BYTE_TS_clear = 0, AO_BYTE_TS_set = 0xff} AO_BYTE_TS_val;\r\n#define AO_TS_VAL_t AO_BYTE_TS_val\r\n#define AO_TS_CLEAR AO_BYTE_TS_clear\r\n#define AO_TS_SET AO_BYTE_TS_set\r\n\r\n#define AO_CHAR_TS_T 1\r\n\r\n\r\n\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/atomic_ops/atomic_ops.h", "content": "/*\r\n * Copyright (c) 2003 Hewlett-Packard Development Company, L.P.\r\n *\r\n * Permission is hereby granted, free of charge, to any person obtaining a copy\r\n * of this software and associated documentation files (the \"Software\"), to deal\r\n * in the Software without restriction, including without limitation the rights\r\n * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\r\n * copies of the Software, and to permit persons to whom the Software is\r\n * furnished to do so, subject to the following conditions:\r\n *\r\n * The above copyright notice and this permission notice shall be included in\r\n * all copies or substantial portions of the Software.\r\n *\r\n * THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\r\n * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\r\n * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\r\n * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\r\n * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\r\n * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\r\n * SOFTWARE.\r\n */\r\n\r\n#ifndef ATOMIC_OPS_H\r\n\r\n#define ATOMIC_OPS_H\r\n\r\n#include \r\n#include \r\n\r\n/* We define various atomic operations on memory in a */\r\n/* machine-specific way. Unfortunately, this is complicated */\r\n/* by the fact that these may or may not be combined with */\r\n/* various memory barriers. Thus the actual operations we */\r\n/* define have the form AO__, for all */\r\n/* plausible combinations of and . */\r\n/* This of course results in a mild combinatorial explosion. */\r\n/* To deal with it, we try to generate derived */\r\n/* definitions for as many of the combinations as we can, as */\r\n/* automatically as possible. */\r\n/* */\r\n/* Our assumption throughout is that the programmer will */\r\n/* specify the least demanding operation and memory barrier */\r\n/* that will guarantee correctness for the implementation. */\r\n/* Our job is to find the least expensive way to implement it */\r\n/* on the applicable hardware. In many cases that will */\r\n/* involve, for example, a stronger memory barrier, or a */\r\n/* combination of hardware primitives. */\r\n/* */\r\n/* Conventions: */\r\n/* \"plain\" atomic operations are not guaranteed to include */\r\n/* a barrier. The suffix in the name specifies the barrier */\r\n/* type. Suffixes are: */\r\n/* _release: Earlier operations may not be delayed past it. */\r\n/* _acquire: Later operations may not move ahead of it. */\r\n/* _read: Subsequent reads must follow this operation and */\r\n/* preceding reads. */\r\n/* _write: Earlier writes precede both this operation and */\r\n/* later writes. */\r\n/* _full: Ordered with respect to both earlier and later memops.*/\r\n/* _release_write: Ordered with respect to earlier writes. */\r\n/* _acquire_read: Ordered with respect to later reads. */\r\n/* */\r\n/* Currently we try to define the following atomic memory */\r\n/* operations, in combination with the above barriers: */\r\n/* AO_nop */\r\n/* AO_load */\r\n/* AO_store */\r\n/* AO_test_and_set (binary) */\r\n/* AO_fetch_and_add */\r\n/* AO_fetch_and_add1 */\r\n/* AO_fetch_and_sub1 */\r\n/* AO_or */\r\n/* AO_compare_and_swap */\r\n/* */\r\n/* Note that atomicity guarantees are valid only if both */\r\n/* readers and writers use AO_ operations to access the */\r\n/* shared value, while ordering constraints are intended to */\r\n/* apply all memory operations. If a location can potentially */\r\n/* be accessed simultaneously from multiple threads, and one of */\r\n/* those accesses may be a write access, then all such */\r\n/* accesses to that location should be through AO_ primitives. */\r\n/* However if AO_ operations enforce sufficient ordering to */\r\n/* ensure that a location x cannot be accessed concurrently, */\r\n/* or can only be read concurrently, then x can be accessed */\r\n/* via ordinary references and assignments. */\r\n/* */\r\n/* Compare_and_exchange takes an address and an expected old */\r\n/* value and a new value, and returns an int. Nonzero */\r\n/* indicates that it succeeded. */\r\n/* Test_and_set takes an address, atomically replaces it by */\r\n/* AO_TS_SET, and returns the prior value. */\r\n/* An AO_TS_t location can be reset with the */\r\n/* AO_CLEAR macro, which normally uses AO_store_release. */\r\n/* AO_fetch_and_add takes an address and an AO_t increment */\r\n/* value. The AO_fetch_and_add1 and AO_fetch_and_sub1 variants */\r\n/* are provided, since they allow faster implementations on */\r\n/* some hardware. AO_or atomically ors an AO_t value into a */\r\n/* memory location, but does not provide access to the original.*/\r\n/* */\r\n/* We expect this list to grow slowly over time. */\r\n/* */\r\n/* Note that AO_nop_full is a full memory barrier. */\r\n/* */\r\n/* Note that if some data is initialized with */\r\n/* data.x = ...; data.y = ...; ... */\r\n/* AO_store_release_write(&data_is_initialized, 1) */\r\n/* then data is guaranteed to be initialized after the test */\r\n/* if (AO_load_release_read(&data_is_initialized)) ... */\r\n/* succeeds. Furthermore, this should generate near-optimal */\r\n/* code on all common platforms. */\r\n/* */\r\n/* All operations operate on unsigned AO_t, which */\r\n/* is the natural word size, and usually unsigned long. */\r\n/* It is possible to check whether a particular operation op */\r\n/* is available on a particular platform by checking whether */\r\n/* AO_HAVE_op is defined. We make heavy use of these macros */\r\n/* internally. */\r\n\r\n/* The rest of this file basically has three sections: */\r\n/* */\r\n/* Some utility and default definitions. */\r\n/* */\r\n/* The architecture dependent section: */\r\n/* This defines atomic operations that have direct hardware */\r\n/* support on a particular platform, mostly by including the */\r\n/* appropriate compiler- and hardware-dependent file. */\r\n/* */\r\n/* The synthesis section: */\r\n/* This tries to define other atomic operations in terms of */\r\n/* those that are explicitly available on the platform. */\r\n/* This section is hardware independent. */\r\n/* We make no attempt to synthesize operations in ways that */\r\n/* effectively introduce locks, except for the debugging/demo */\r\n/* pthread-based implementation at the beginning. A more */\r\n/* realistic implementation that falls back to locks could be */\r\n/* added as a higher layer. But that would sacrifice */\r\n/* usability from signal handlers. */\r\n/* The synthesis section is implemented almost entirely in */\r\n/* atomic_ops_generalize.h. */\r\n\r\n/* Some common defaults. Overridden for some architectures. */\r\n#define AO_t size_t\r\n\r\n/* The test_and_set primitive returns an AO_TS_VAL_t value. */\r\n/* AO_TS_t is the type of an in-memory test-and-set location. */\r\n\r\n#define AO_TS_INITIALIZER (AO_t)AO_TS_CLEAR\r\n\r\n/* Platform-dependent stuff: */\r\n#if defined(__GNUC__) || defined(_MSC_VER) || defined(__INTEL_COMPILER) \\\r\n || defined(__DMC__) || defined(__WATCOMC__)\r\n# define AO_INLINE static __inline\r\n#elif defined(__sun)\r\n# define AO_INLINE static inline\r\n#else\r\n# define AO_INLINE static\r\n#endif\r\n\r\n#if defined(__GNUC__) && !defined(__INTEL_COMPILER)\r\n# define AO_compiler_barrier() __asm__ __volatile__(\"\" : : : \"memory\")\r\n#elif defined(_MSC_VER) || defined(__DMC__) || defined(__BORLANDC__) \\\r\n || defined(__WATCOMC__)\r\n# if defined(_AMD64_) || defined(_M_X64) || _MSC_VER >= 1400\r\n# if defined(_WIN32_WCE)\r\n/* # include */\r\n# elif defined(_MSC_VER)\r\n# include \r\n# endif\r\n# pragma intrinsic(_ReadWriteBarrier)\r\n# define AO_compiler_barrier() _ReadWriteBarrier()\r\n /* We assume this does not generate a fence instruction. */\r\n /* The documentation is a bit unclear. */\r\n# else\r\n# define AO_compiler_barrier() __asm { }\r\n /* The preceding implementation may be preferable here too. */\r\n /* But the documentation warns about VC++ 2003 and earlier. */\r\n# endif\r\n#elif defined(__INTEL_COMPILER)\r\n# define AO_compiler_barrier() __memory_barrier() /* Too strong? IA64-only? */\r\n#elif defined(_HPUX_SOURCE)\r\n# if defined(__ia64)\r\n# include \r\n# define AO_compiler_barrier() _Asm_sched_fence()\r\n# else\r\n /* FIXME - We dont know how to do this. This is a guess. */\r\n /* And probably a bad one. */\r\n static volatile int AO_barrier_dummy;\r\n# define AO_compiler_barrier() AO_barrier_dummy = AO_barrier_dummy\r\n# endif\r\n#else\r\n /* We conjecture that the following usually gives us the right */\r\n /* semantics or an error. */\r\n# define AO_compiler_barrier() asm(\"\")\r\n#endif\r\n\r\n#if defined(AO_USE_PTHREAD_DEFS)\r\n# include \"atomic_ops/sysdeps/generic_pthread.h\"\r\n#endif /* AO_USE_PTHREAD_DEFS */\r\n\r\n#if (defined(__CC_ARM) || defined(__ARMCC__)) && !defined(__GNUC__) \\\r\n && !defined(AO_USE_PTHREAD_DEFS)\r\n# include \"atomic_ops/sysdeps/armcc/arm_v6.h\"\r\n# define AO_GENERALIZE_TWICE\r\n#endif\r\n\r\n#if defined(__GNUC__) && !defined(AO_USE_PTHREAD_DEFS) \\\r\n && !defined(__INTEL_COMPILER)\r\n# if defined(__i386__)\r\n /* We don't define AO_USE_SYNC_CAS_BUILTIN for x86 here because */\r\n /* it might require specifying additional options (like -march) */\r\n /* or additional link libraries (if -march is not specified). */\r\n# include \"atomic_ops/sysdeps/gcc/x86.h\"\r\n# endif /* __i386__ */\r\n# if defined(__x86_64__)\r\n# if __GNUC__ > 4 || (__GNUC__ == 4 && __GNUC_MINOR__ >= 2)\r\n /* It is safe to use __sync CAS built-in on this architecture. */\r\n# define AO_USE_SYNC_CAS_BUILTIN\r\n# endif\r\n# include \"atomic_ops/sysdeps/gcc/x86_64.h\"\r\n# endif /* __x86_64__ */\r\n# if defined(__ia64__)\r\n# include \"atomic_ops/sysdeps/gcc/ia64.h\"\r\n# define AO_GENERALIZE_TWICE\r\n# endif /* __ia64__ */\r\n# if defined(__hppa__)\r\n# include \"atomic_ops/sysdeps/gcc/hppa.h\"\r\n# define AO_CAN_EMUL_CAS\r\n# endif /* __hppa__ */\r\n# if defined(__alpha__)\r\n# include \"atomic_ops/sysdeps/gcc/alpha.h\"\r\n# define AO_GENERALIZE_TWICE\r\n# endif /* __alpha__ */\r\n# if defined(__s390__)\r\n# include \"atomic_ops/sysdeps/gcc/s390.h\"\r\n# endif /* __s390__ */\r\n# if defined(__sparc__)\r\n# include \"atomic_ops/sysdeps/gcc/sparc.h\"\r\n# define AO_CAN_EMUL_CAS\r\n# endif /* __sparc__ */\r\n# if defined(__m68k__)\r\n# include \"atomic_ops/sysdeps/gcc/m68k.h\"\r\n# endif /* __m68k__ */\r\n# if defined(__powerpc__) || defined(__ppc__) || defined(__PPC__) \\\r\n || defined(__powerpc64__) || defined(__ppc64__)\r\n# include \"atomic_ops/sysdeps/gcc/powerpc.h\"\r\n# endif /* __powerpc__ */\r\n# if defined(__arm__) && !defined(AO_USE_PTHREAD_DEFS)\r\n# include \"atomic_ops/sysdeps/gcc/arm.h\"\r\n# define AO_CAN_EMUL_CAS\r\n# endif /* __arm__ */\r\n# if defined(__cris__) || defined(CRIS)\r\n# include \"atomic_ops/sysdeps/gcc/cris.h\"\r\n# define AO_GENERALIZE_TWICE\r\n# endif\r\n# if defined(__mips__)\r\n# include \"atomic_ops/sysdeps/gcc/mips.h\"\r\n# endif /* __mips__ */\r\n# if defined(__sh__) || defined(SH4)\r\n# include \"atomic_ops/sysdeps/gcc/sh.h\"\r\n# define AO_CAN_EMUL_CAS\r\n# endif /* __sh__ */\r\n# if defined(__avr32__)\r\n# include \"atomic_ops/sysdeps/gcc/avr32.h\"\r\n# endif\r\n#endif /* __GNUC__ && !AO_USE_PTHREAD_DEFS */\r\n\r\n#if (defined(__IBMC__) || defined(__IBMCPP__)) && !defined(__GNUC__) \\\r\n && !defined(AO_USE_PTHREAD_DEFS)\r\n# if defined(__powerpc__) || defined(__powerpc) || defined(__ppc__) \\\r\n || defined(__PPC__) || defined(_M_PPC) || defined(_ARCH_PPC) \\\r\n || defined(_ARCH_PWR)\r\n# include \"atomic_ops/sysdeps/ibmc/powerpc.h\"\r\n# define AO_GENERALIZE_TWICE\r\n# endif\r\n#endif\r\n\r\n#if defined(__INTEL_COMPILER) && !defined(AO_USE_PTHREAD_DEFS)\r\n# if defined(__ia64__)\r\n# include \"atomic_ops/sysdeps/icc/ia64.h\"\r\n# define AO_GENERALIZE_TWICE\r\n# endif\r\n# if defined(__GNUC__)\r\n /* Intel Compiler in GCC compatible mode */\r\n# if defined(__i386__)\r\n# include \"atomic_ops/sysdeps/gcc/x86.h\"\r\n# endif /* __i386__ */\r\n# if defined(__x86_64__)\r\n# if __INTEL_COMPILER > 1110\r\n# define AO_USE_SYNC_CAS_BUILTIN\r\n# endif\r\n# include \"atomic_ops/sysdeps/gcc/x86_64.h\"\r\n# endif /* __x86_64__ */\r\n# endif\r\n#endif\r\n\r\n#if defined(_HPUX_SOURCE) && !defined(__GNUC__) && !defined(AO_USE_PTHREAD_DEFS)\r\n# if defined(__ia64)\r\n# include \"atomic_ops/sysdeps/hpc/ia64.h\"\r\n# define AO_GENERALIZE_TWICE\r\n# else\r\n# include \"atomic_ops/sysdeps/hpc/hppa.h\"\r\n# define AO_CAN_EMUL_CAS\r\n# endif\r\n#endif\r\n\r\n#if defined(_MSC_VER) || defined(__DMC__) || defined(__BORLANDC__) \\\r\n || (defined(__WATCOMC__) && defined(__NT__))\r\n# if defined(_AMD64_) || defined(_M_X64)\r\n# include \"atomic_ops/sysdeps/msftc/x86_64.h\"\r\n# elif defined(_M_IX86) || defined(x86)\r\n# include \"atomic_ops/sysdeps/msftc/x86.h\"\r\n# elif defined(_M_ARM) || defined(ARM) || defined(_ARM_)\r\n# include \"atomic_ops/sysdeps/msftc/arm.h\"\r\n# define AO_GENERALIZE_TWICE\r\n# endif\r\n#endif\r\n\r\n#if defined(__sun) && !defined(__GNUC__) && !defined(AO_USE_PTHREAD_DEFS)\r\n /* Note: use -DAO_USE_PTHREAD_DEFS if Sun CC does not handle inline asm. */\r\n# if defined(__i386)\r\n# include \"atomic_ops/sysdeps/sunc/x86.h\"\r\n# endif /* __i386 */\r\n# if defined(__x86_64) || defined(__amd64)\r\n# include \"atomic_ops/sysdeps/sunc/x86_64.h\"\r\n# endif /* __x86_64 */\r\n#endif\r\n\r\n#if !defined(__GNUC__) && (defined(sparc) || defined(__sparc)) \\\r\n && !defined(AO_USE_PTHREAD_DEFS)\r\n# include \"atomic_ops/sysdeps/sunc/sparc.h\"\r\n# define AO_CAN_EMUL_CAS\r\n#endif\r\n\r\n#if defined(AO_REQUIRE_CAS) && !defined(AO_HAVE_compare_and_swap) \\\r\n && !defined(AO_HAVE_compare_and_swap_full) \\\r\n && !defined(AO_HAVE_compare_and_swap_acquire)\r\n# if defined(AO_CAN_EMUL_CAS)\r\n# include \"atomic_ops/sysdeps/emul_cas.h\"\r\n# else\r\n# error Cannot implement AO_compare_and_swap_full on this architecture.\r\n# endif\r\n#endif /* AO_REQUIRE_CAS && !AO_HAVE_compare_and_swap ... */\r\n\r\n/* The most common way to clear a test-and-set location */\r\n/* at the end of a critical section. */\r\n#if AO_AO_TS_T && !defined(AO_CLEAR)\r\n# define AO_CLEAR(addr) AO_store_release((AO_TS_t *)(addr), AO_TS_CLEAR)\r\n#endif\r\n#if AO_CHAR_TS_T && !defined(AO_CLEAR)\r\n# define AO_CLEAR(addr) AO_char_store_release((AO_TS_t *)(addr), AO_TS_CLEAR)\r\n#endif\r\n\r\n/* The generalization section. */\r\n#if !defined(AO_GENERALIZE_TWICE) && defined(AO_CAN_EMUL_CAS) \\\r\n && !defined(AO_HAVE_compare_and_swap_full)\r\n# define AO_GENERALIZE_TWICE\r\n#endif\r\n\r\n/* Theoretically we should repeatedly include atomic_ops_generalize.h. */\r\n/* In fact, we observe that this converges after a small fixed number */\r\n/* of iterations, usually one. */\r\n#include \"atomic_ops/generalize.h\"\r\n#ifdef AO_GENERALIZE_TWICE\r\n# include \"atomic_ops/generalize.h\"\r\n#endif\r\n\r\n/* For compatibility with version 0.4 and earlier */\r\n#define AO_TS_T AO_TS_t\r\n#define AO_T AO_t\r\n#define AO_TS_VAL AO_TS_VAL_t\r\n\r\n#endif /* ATOMIC_OPS_H */\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/dcss/dcss_plus.h", "content": "/* \r\n * File: dcss_plus.h\r\n * Author: Maya Arbel-Raviv\r\n *\r\n * Created on May 1, 2017, 10:42 AM\r\n */\r\n\r\n#ifndef DCSS_PLUS_H\r\n#define DCSS_PLUS_H\r\n\r\n#include \r\n#include \r\n#include \r\n#include \"descriptors.h\"\r\n\r\n#define dcssptagptr_t uintptr_t\r\n#define dcsspptr_t dcsspdesc_t *\r\n#define casword_t intptr_t\r\n\r\n#define DCSSP_STATE_UNDECIDED 0\r\n#define DCSSP_STATE_SUCCEEDED 4\r\n#define DCSSP_STATE_FAILED 8\r\n\r\n#define DCSSP_LEFTSHIFT 1\r\n\r\n#define DCSSP_IGNORED_RETVAL -1\r\n#define DCSSP_SUCCESS 0\r\n#define DCSSP_FAILED_ADDR1 1 \r\n#define DCSSP_FAILED_ADDR2 2\r\n\r\n#define MAX_PAYLOAD_PTRS 6\r\n\r\nstruct dcsspresult_t {\r\n int status;\r\n casword_t failed_val;\r\n};\r\n\r\ntemplate \r\nclass dcsspdesc_t {\r\npublic:\r\n volatile mutables_t mutables;\r\n casword_t volatile * volatile addr1;\r\n casword_t volatile old1;\r\n casword_t volatile * volatile addr2;\r\n casword_t volatile old2;\r\n casword_t volatile new2;\r\n PAYLOAD_T volatile payload1[MAX_PAYLOAD_PTRS+1];\r\n PAYLOAD_T volatile payload2[MAX_PAYLOAD_PTRS+1];\r\n const static int size = sizeof(mutables)+sizeof(addr1)+sizeof(old1)+sizeof(addr2)+sizeof(old2)+sizeof(new2)+sizeof(PAYLOAD_T)*(MAX_PAYLOAD_PTRS+1)+sizeof(PAYLOAD_T)*(MAX_PAYLOAD_PTRS+1);\r\n char padding[PREFETCH_SIZE_BYTES+(((64<<10)-size%64)%64)]; // add padding to prevent false sharing\r\n} __attribute__ ((aligned(64)));\r\n\r\ntemplate \r\nclass dcsspProvider {\r\n /**\r\n * Data definitions\r\n */\r\nprivate:\r\n // descriptor reduction algorithm\r\n #define DCSSP_MUTABLES_OFFSET_STATE 0\r\n #define DCSSP_MUTABLES_MASK_STATE 0xf\r\n #define DCSSP_MUTABLES_NEW(mutables) \\\r\n ((((mutables)&MASK_SEQ)+(1< dcsspDescriptors[LAST_TID+1] __attribute__ ((aligned(64)));\r\n char __padding_desc3[PREFETCH_SIZE_BYTES];\r\n\r\npublic:\r\n#ifdef USE_DEBUGCOUNTERS\r\n debugCounter * dcsspHelpCounter;\r\n#endif\r\n const int NUM_PROCESSES;\r\n \r\n /**\r\n * Function declarations\r\n */\r\n dcsspProvider(const int numProcesses);\r\n ~dcsspProvider();\r\n void initThread(const int tid);\r\n void deinitThread(const int tid);\r\n void writePtr(casword_t volatile * addr, casword_t val); // use for addresses that might have been modified by DCSSP (ONLY GOOD FOR INITIALIZING, CANNOT DEAL WITH CONCURRENT DCSSP OPERATIONS.)\r\n void writeVal(casword_t volatile * addr, casword_t val); // use for addresses that might have been modified by DCSSP (ONLY GOOD FOR INITIALIZING, CANNOT DEAL WITH CONCURRENT DCSSP OPERATIONS.)\r\n casword_t readPtr(const int tid, casword_t volatile * addr); // use for addresses that might have been modified by DCSSP\r\n casword_t readVal(const int tid, casword_t volatile * addr); // use for addresses that might have been modified by DCSSP\r\n inline dcsspresult_t dcsspPtr(const int tid, casword_t * addr1, casword_t old1, casword_t * addr2, casword_t old2, casword_t new2, PAYLOAD_T * const payload1, PAYLOAD_T * const payload2); // use when addr2 is a pointer, or another type that does not use its least significant bit\r\n inline dcsspresult_t dcsspVal(const int tid, casword_t * addr1, casword_t old1, casword_t * addr2, casword_t old2, casword_t new2, PAYLOAD_T * const payload1, PAYLOAD_T * const payload2); // use when addr2 uses its least significant bit, but does not use its most significant but\r\n void discardPayloads(const int tid);\r\n void debugPrint();\r\n \r\n tagptr_t getDescriptorTagptr(const int otherTid);\r\n dcsspptr_t getDescriptorPtr(tagptr_t tagptr);\r\n bool getDescriptorSnapshot(tagptr_t tagptr, dcsspptr_t const dest);\r\n void helpProcess(const int tid, const int otherTid);\r\nprivate:\r\n casword_t dcsspRead(const int tid, casword_t volatile * addr);\r\n inline dcsspresult_t dcsspHelp(const int tid, dcssptagptr_t tagptr, dcsspptr_t snapshot, bool helpingOther);\r\n void dcsspHelpOther(const int tid, dcssptagptr_t tagptr);\r\n};\r\n\r\n#endif /* DCSS_PLUS_H */\r\n\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/dcss/dcss_plus_impl.h", "content": "/* \r\n * File: dcss_plus_impl.h\r\n * Author: Maya Arbel-Raviv\r\n *\r\n * Created on May 1, 2017, 10:52 AM\r\n */\r\n\r\n#ifndef DCSS_PLUS_IMPL_H\r\n#define DCSS_PLUS_IMPL_H\r\n\r\n#include \"dcss_plus.h\"\r\n#include \r\n#include \r\n#include \r\nusing namespace std;\r\n\r\n#define BOOL_CAS __sync_bool_compare_and_swap\r\n#define VAL_CAS __sync_val_compare_and_swap\r\n\r\n#define DCSSP_TAGBIT 0x1\r\n\r\nstatic bool isDcssp(casword_t val) {\r\n return (val & DCSSP_TAGBIT);\r\n}\r\n\r\ntemplate \r\ndcsspresult_t dcsspProvider::dcsspHelp(const int tid, dcssptagptr_t tagptr, dcsspptr_t snapshot, bool helpingOther) {\r\n // figure out what the state should be\r\n casword_t state = DCSSP_STATE_FAILED;\r\n\r\n SOFTWARE_BARRIER;\r\n casword_t val1 = *(snapshot->addr1);\r\n SOFTWARE_BARRIER;\r\n \r\n //DELAY_UP_TO(1000);\r\n if (val1 == snapshot->old1) { // linearize here(?)\r\n state = DCSSP_STATE_SUCCEEDED;\r\n }\r\n \r\n // try to cas the state to the appropriate value\r\n dcsspptr_t ptr = TAGPTR_UNPACK_PTR(dcsspDescriptors,tagptr);\r\n casword_t retval;\r\n bool failedBit;\r\n MUTABLES_VAL_CAS_FIELD(failedBit, retval, ptr->mutables, snapshot->mutables, DCSSP_STATE_UNDECIDED, state, DCSSP_MUTABLES_MASK_STATE, DCSSP_MUTABLES_OFFSET_STATE); \r\n if (failedBit) return {DCSSP_IGNORED_RETVAL,0}; // failed to access the descriptor: we must be helping another process complete its operation, so we will NOT use this return value!\r\n \r\n // TODO: do we do the announcement here? what will be announced exactly? do we let the user provide a pointer/value to announce as an argument to dcssp? do we need to provide an operation to retrieve the current announcement for a given process?\r\n \r\n // finish the operation based on the descriptor's state\r\n if ((retval == DCSSP_STATE_UNDECIDED && state == DCSSP_STATE_SUCCEEDED) // if we changed the state to succeeded OR\r\n || retval == DCSSP_STATE_SUCCEEDED) { // if someone else changed the state to succeeded\r\n// if (state == DCSSP_STATE_FAILED) DELAY_UP_TO(1000);\r\n assert(helpingOther || ((snapshot->mutables & DCSSP_MUTABLES_MASK_STATE) >> DCSSP_MUTABLES_OFFSET_STATE) == DCSSP_STATE_SUCCEEDED);\r\n BOOL_CAS(snapshot->addr2, (casword_t) tagptr, snapshot->new2); \r\n return {DCSSP_SUCCESS,0};\r\n } else { // either we or someone else changed the state to failed\r\n assert((retval == DCSSP_STATE_UNDECIDED && state == DCSSP_STATE_FAILED)\r\n || retval == DCSSP_STATE_FAILED);\r\n assert(helpingOther || ((snapshot->mutables & DCSSP_MUTABLES_MASK_STATE) >> DCSSP_MUTABLES_OFFSET_STATE) == DCSSP_STATE_FAILED);\r\n BOOL_CAS(snapshot->addr2, (casword_t) tagptr, snapshot->old2);\r\n// if (state == DCSSP_STATE_FAILED) DELAY_UP_TO(1000);\r\n return {DCSSP_FAILED_ADDR1,val1};\r\n }\r\n}\r\n\r\ntemplate \r\nvoid dcsspProvider::dcsspHelpOther(const int tid, dcssptagptr_t tagptr) {\r\n const int otherTid = TAGPTR_UNPACK_TID(tagptr);\r\n assert(otherTid >= 0 && otherTid < NUM_PROCESSES);\r\n dcsspdesc_t newSnapshot;\r\n const int sz = dcsspdesc_t::size;\r\n assert((((tagptr & MASK_SEQ) >> OFFSET_SEQ) & 1) == 1);\r\n if (DESC_SNAPSHOT(dcsspdesc_t, dcsspDescriptors, &newSnapshot, tagptr, sz)) {\r\n dcsspHelp(tid, tagptr, &newSnapshot, true);\r\n } else {\r\n //TRACE COUTATOMICTID(\"helpOther unable to get snapshot of \"<\r\ninline\r\ntagptr_t dcsspProvider::getDescriptorTagptr(const int otherTid) {\r\n dcsspptr_t ptr = &dcsspDescriptors[otherTid];\r\n tagptr_t tagptr = TAGPTR_NEW(otherTid, ptr->mutables, DCSSP_TAGBIT);\r\n if ((UNPACK_SEQ(tagptr) & 1) == 0) {\r\n // descriptor is being initialized! essentially,\r\n // we can think of there being NO ongoing operation,\r\n // so we can imagine we return NULL = no descriptor.\r\n return (tagptr_t) NULL;\r\n }\r\n return tagptr;\r\n}\r\n\r\ntemplate \r\ninline\r\ndcsspptr_t dcsspProvider::getDescriptorPtr(tagptr_t tagptr) {\r\n return TAGPTR_UNPACK_PTR(dcsspDescriptors, tagptr);\r\n}\r\n\r\ntemplate \r\ninline\r\nbool dcsspProvider::getDescriptorSnapshot(tagptr_t tagptr, dcsspptr_t const dest) {\r\n if (tagptr == (tagptr_t) NULL) return false;\r\n return DESC_SNAPSHOT(dcsspdesc_t, dcsspDescriptors, dest, tagptr, dcsspdesc_t::size);\r\n}\r\n\r\ntemplate \r\ninline\r\nvoid dcsspProvider::helpProcess(const int tid, const int otherTid) {\r\n tagptr_t tagptr = getDescriptorTagptr(otherTid);\r\n if (tagptr != (tagptr_t) NULL) dcsspHelpOther(tid, tagptr);\r\n}\r\n\r\ntemplate \r\nvoid dcsspProvider::discardPayloads(const int tid) {\r\n SOFTWARE_BARRIER;\r\n dcssptagptr_t tagptr = getDescriptorTagptr(tid);\r\n dcsspptr_t ptr = getDescriptorPtr(tagptr);\r\n ptr->payload1[0] = NULL;\r\n ptr->payload2[0] = NULL;\r\n SOFTWARE_BARRIER;\r\n}\r\n\r\ntemplate \r\ndcsspresult_t dcsspProvider::dcsspVal(const int tid, casword_t * addr1, casword_t old1, casword_t * addr2, casword_t old2, casword_t new2, PAYLOAD_T * const payload1, PAYLOAD_T * const payload2) {\r\n return dcsspPtr(tid, addr1, old1, addr2, old2 << DCSSP_LEFTSHIFT , new2 << DCSSP_LEFTSHIFT, payload1, payload2);\r\n}\r\n\r\ntemplate \r\ndcsspresult_t dcsspProvider::dcsspPtr(const int tid, casword_t * addr1, casword_t old1, casword_t * addr2, casword_t old2, casword_t new2, PAYLOAD_T * const payload1, PAYLOAD_T * const payload2) {\r\n // create dcssp descriptor\r\n dcsspptr_t ptr = DESC_NEW(dcsspDescriptors, DCSSP_MUTABLES_NEW, tid);\r\n assert((((dcsspDescriptors[tid].mutables & MASK_SEQ) >> OFFSET_SEQ) & 1) == 0);\r\n ptr->addr1 = addr1;\r\n ptr->old1 = old1;\r\n ptr->addr2 = addr2;\r\n ptr->old2 = old2;\r\n ptr->new2 = new2;\r\n \r\n // add payload1 and payload2 to the dcssp descriptor\r\n int i;\r\n for (i=0;payload1[i];++i) {\r\n ptr->payload1[i] = payload1[i];\r\n assert(i < MAX_PAYLOAD_PTRS);\r\n }\r\n ptr->payload1[i] = NULL;\r\n for (i=0;payload2[i];++i) {\r\n ptr->payload2[i] = payload2[i];\r\n assert(i < MAX_PAYLOAD_PTRS);\r\n }\r\n ptr->payload2[i] = NULL;\r\n DESC_INITIALIZED(dcsspDescriptors, tid);\r\n \r\n // create tagptr\r\n assert((((dcsspDescriptors[tid].mutables & MASK_SEQ) >> OFFSET_SEQ) & 1) == 1);\r\n tagptr_t tagptr = TAGPTR_NEW(tid, ptr->mutables, DCSSP_TAGBIT);\r\n \r\n // perform the dcssp operation described by our descriptor\r\n casword_t r;\r\n do {\r\n assert(!isDcssp(ptr->old2));\r\n assert(isDcssp(tagptr));\r\n r = VAL_CAS(ptr->addr2, ptr->old2, (casword_t) tagptr);\r\n if (isDcssp(r)) {\r\n#ifdef USE_DEBUGCOUNTERS\r\n this->dcsspHelpCounter->inc(tid);\r\n#endif\r\n dcsspHelpOther(tid, (dcssptagptr_t) r);\r\n }\r\n } while (isDcssp(r));\r\n if (r == ptr->old2){\r\n// DELAY_UP_TO(1000);\r\n return dcsspHelp(tid, tagptr, ptr, false); // finish our own operation \r\n } \r\n return {DCSSP_FAILED_ADDR2,r};//DCSSP_FAILED_ADDR2;\r\n}\r\n\r\ntemplate \r\ncasword_t dcsspProvider::dcsspRead(const int tid, casword_t volatile * addr) {\r\n casword_t r;\r\n while (1) {\r\n r = *addr;\r\n if (isDcssp(r)) {\r\n#ifdef USE_DEBUGCOUNTERS\r\n this->dcsspHelpCounter->inc(tid);\r\n#endif\r\n dcsspHelpOther(tid, (dcssptagptr_t) r);\r\n } else {\r\n return r;\r\n }\r\n }\r\n}\r\n\r\ntemplate \r\ndcsspProvider::dcsspProvider(const int numProcesses) : NUM_PROCESSES(numProcesses) {\r\n#ifdef USE_DEBUGCOUNTERS\r\n dcsspHelpCounter = new debugCounter(NUM_PROCESSES);\r\n#endif\r\n DESC_INIT_ALL(dcsspDescriptors, DCSSP_MUTABLES_NEW, NUM_PROCESSES);\r\n for (int tid=0;tid\r\ndcsspProvider::~dcsspProvider() {\r\n#ifdef USE_DEBUGCOUNTERS\r\n delete dcsspHelpCounter;\r\n#endif\r\n}\r\n\r\ntemplate \r\ncasword_t dcsspProvider::readPtr(const int tid, casword_t volatile * addr) {\r\n casword_t r;\r\n r = dcsspRead(tid, addr);\r\n return r;\r\n}\r\n\r\ntemplate \r\ncasword_t dcsspProvider::readVal(const int tid, casword_t volatile * addr) {\r\n return ((casword_t) readPtr(tid, addr))>>DCSSP_LEFTSHIFT;\r\n}\r\n\r\ntemplate \r\nvoid dcsspProvider::writePtr(casword_t volatile * addr, casword_t ptr) {\r\n //assert((*addr & DCSSP_TAGBIT) == 0);\r\n assert((ptr & DCSSP_TAGBIT) == 0);\r\n *addr = ptr;\r\n}\r\n\r\ntemplate \r\nvoid dcsspProvider::writeVal(casword_t volatile * addr, casword_t val) {\r\n writePtr(addr, val<\r\nvoid dcsspProvider::initThread(const int tid) {}\r\n\r\ntemplate \r\nvoid dcsspProvider::deinitThread(const int tid) {}\r\n\r\ntemplate \r\nvoid dcsspProvider::debugPrint() {\r\n#ifdef USE_DEBUGCOUNTERS\r\n cout<<\"dcssp helping : \"<dcsspHelpCounter->getTotal()<\r\n#include \r\n#include \r\n#include \"dcss_plus_impl.h\"\r\n\r\nusing namespace std;\r\n\r\n#define NUM_OPS 10000000\r\n#define INCREMENT 1\r\n\r\n#define FALSE_SHARING_ULL_FACTOR 24\r\n#define FALSE_SHARING_PAD_BYTES 192\r\n\r\n#define COUNTER(tid) (counters[(tid)*FALSE_SHARING_ULL_FACTOR])\r\n\r\nint numProcesses = 0;\r\nvolatile unsigned long long counters[MAX_TID_POW2*FALSE_SHARING_ULL_FACTOR];\r\nvolatile char padding[FALSE_SHARING_PAD_BYTES];\r\nvolatile unsigned long long faa;\r\n\r\nvolatile bool start;\r\nvolatile int running; // number of threads that are running\r\ndcsspProvider * prov;\r\n\r\n#ifndef KERNEL\r\n#define KERNEL test_kernel1\r\n#endif\r\n\r\n#ifndef VALIDATE\r\n#define VALIDATE validate1\r\n#endif\r\n\r\n//#define GET_FAA_FOR_TID(tid) ((faa >> ((tid)*(62/numProcesses))) & (numProcesses == 1 ? 0xffffffffffffffffULL : ((1ULL<<(62/numProcesses))-1)))\r\n\r\nvoid * test_kernel1(void * arg) {\r\n const int tid = *((int *) arg);\r\n //const unsigned long long numOps = min(1ULL<<20, 1ULL<<(62/numProcesses)-1);\r\n //const unsigned long long increment = 1ULL<<(tid*(62/numProcesses));\r\n prov->initThread(tid);\r\n __sync_fetch_and_add(&running, 1);\r\n while (!start) { __sync_synchronize(); }\r\n \r\n //COUTATOMICTID(\"performing \"<readVal(tid,(casword_t*)&(COUNTER(tid)));\r\n casword_t newval = (casword_t) oldval+1;\r\n if (DCSSP_SUCCESS == prov->dcsspVal(tid, (casword_t *) &faa, (casword_t) faa, (casword_t *) &COUNTER(tid), oldval, newval, deletedNodes)) {\r\n ++numSucc;\r\n __sync_fetch_and_add(&faa, INCREMENT);\r\n }\r\n#else\r\n ++numSucc;\r\n ++COUNTER(tid);\r\n __sync_fetch_and_add(&faa, INCREMENT);\r\n#endif\r\n }\r\n \r\n prov->deinitThread(tid);\r\n}\r\n\r\nbool validate1() {\r\n // compute checksum\r\n bool good = true;\r\n for (int i=0;ireadVal(i,(casword_t*)&(COUNTER(i)));\r\n if (c != NUM_OPS) {\r\n cout<<\"ERROR: counters[\"<initThread(tid);\r\n __sync_fetch_and_add(&running, 1);\r\n while (!start) { __sync_synchronize(); }\r\n \r\n //COUTATOMICTID(\"performing \"<readVal(tid, (casword_t *) &faa);\r\n casword_t new2 = (casword_t) old2+1;\r\n if (DCSSP_SUCCESS == prov->dcsspVal(tid, (casword_t *) &COUNTER((tid+1)%numProcesses), old1, (casword_t *) &faa, old2, new2, deletedNodes)) {\r\n ++numSucc;\r\n ++COUNTER(tid);\r\n }\r\n }\r\n prov->deinitThread(tid);\r\n}\r\n\r\nbool validate2() {\r\n // compute checksum\r\n bool good = true;\r\n for (int i=0;ireadVal(tid, (casword_t *) &faa);\r\n if (f != NUM_OPS * numProcesses) {\r\n cout<<\"ERROR: faa=\"<.\r\n * these three fields are defined by the TAGPTR_ macros below.\r\n */\r\n\r\n#ifndef WIDTH1_SEQ\r\n #define WIDTH1_SEQ 48\r\n#endif\r\n#define OFFSET1_SEQ 11\r\n#define MASK1_SEQ ((uintptr_t)((1LL<>OFFSET1_SEQ)\r\n\r\n#define TAGPTR1_OFFSET_STALE 0 /* UNUSED */\r\n#define TAGPTR1_OFFSET_TID 1\r\n#define TAGPTR1_MASK_STALE 0x1 /* UNUSED */\r\n#define TAGPTR1_MASK_TID (((1<>TAGPTR1_OFFSET_TID))\r\n#define TAGPTR1_UNPACK_PTR(tagptr) (&DESC1_ARRAY[TAGPTR1_UNPACK_TID((tagptr))])\r\n#define TAGPTR1_NEW(tid, mutables) ((tagptr_t) (((UNPACK1_SEQ(mutables))<>TAGPTR1_OFFSET_TID)\r\n#define TAGPTR1_STATIC_DESC(id) ((tagptr_t) TAGPTR1_NEW(LAST_TID1-1-id, 0))\r\n#define TAGPTR1_DUMMY_DESC(id) ((tagptr_t) TAGPTR1_NEW(LAST_TID1, id<>(offset))\r\n#define MUTABLES1_WRITE_FIELD(fldMutables, snapMutables, val, mask, offset) { \\\r\n mutables_t __v = (fldMutables); \\\r\n while (UNPACK1_SEQ(__v) == UNPACK1_SEQ((snapMutables)) \\\r\n && MUTABLES1_UNPACK_FIELD(__v, (mask), (offset)) != (val) \\\r\n && !__sync_bool_compare_and_swap(&(fldMutables), __v, \\\r\n (__v & ~(mask)) | ((val)<<(offset)))) { \\\r\n __v = (fldMutables); \\\r\n } \\\r\n}\r\n#define MUTABLES1_WRITE_BIT(fldMutables, snapMutables, mask) { \\\r\n mutables_t __v = (fldMutables); \\\r\n while (UNPACK1_SEQ(__v) == UNPACK1_SEQ((snapMutables)) \\\r\n && !(__v&(mask)) \\\r\n && !__sync_bool_compare_and_swap(&(fldMutables), __v, (__v|(mask)))) { \\\r\n __v = (fldMutables); \\\r\n } \\\r\n}\r\n\r\n// WARNING: uses a GCC extension \"({ })\". to get rid of this, use an inline function.\r\n#define DESC1_SNAPSHOT(descDest, tagptr, sz) ({ \\\r\n DESC1_T *__src = TAGPTR1_UNPACK_PTR((tagptr)); \\\r\n memcpy((descDest), __src, (sz)); \\\r\n SOFTWARE_BARRIER; /* prevent compiler from reordering read of __src->mutables before (at least the reading portion of) the memcpy */ \\\r\n (UNPACK1_SEQ(__src->c.mutables) == UNPACK1_SEQ((tagptr))); \\\r\n})\r\n#define DESC1_READ_FIELD(successBit, fldMutables, tagptr, mask, offset) ({ \\\r\n mutables_t __mutables = (fldMutables); \\\r\n successBit = (UNPACK1_SEQ(__mutables) == UNPACK1_SEQ(tagptr)); \\\r\n MUTABLES1_UNPACK_FIELD(__mutables, (mask), (offset)); \\\r\n})\r\n#define DESC1_NEW(tid) &DESC1_ARRAY[(tid)]; { /* note: only the process invoking this following macro can change the sequence# */ \\\r\n SOFTWARE_BARRIER; \\\r\n uintptr_t __v = DESC1_ARRAY[(tid)].c.mutables; \\\r\n/* while (!__sync_bool_compare_and_swap(&DESC1_ARRAY[(tid)].mutables, __v, MUTABLES1_NEW(__v))) { \\\r\n __v = DESC1_ARRAY[(tid)].mutables; \\\r\n } \\\r\n}*/ \\\r\n DESC1_ARRAY[(tid)].c.mutables = MUTABLES1_NEW(__v); \\\r\n /*__sync_synchronize();*/ \\\r\n SOFTWARE_BARRIER; \\\r\n}\r\n#define DESC1_INITIALIZED(tid) \\\r\n SOFTWARE_BARRIER; \\\r\n DESC1_ARRAY[(tid)].c.mutables += (1<.\r\n * these three fields are defined by the TAGPTR_ macros below.\r\n */\r\n\r\n#ifndef WIDTH_SEQ\r\n #define WIDTH_SEQ 48\r\n#endif\r\n#define OFFSET_SEQ 14\r\n#define MASK_SEQ ((uintptr_t)((1LL<>OFFSET_SEQ)\r\n\r\n#define TAGPTR_OFFSET_USER 0\r\n#define TAGPTR_OFFSET_TID 3\r\n#define TAGPTR_MASK_USER ((1<>TAGPTR_OFFSET_TID))\r\n#define TAGPTR_UNPACK_PTR(descArray, tagptr) (&(descArray)[TAGPTR_UNPACK_TID((tagptr))])\r\n#define TAGPTR_NEW(tid, mutables, userBits) ((tagptr_t) (((UNPACK_SEQ(mutables))<>TAGPTR_OFFSET_TID)\r\n#define TAGPTR_STATIC_DESC(id) ((tagptr_t) TAGPTR_NEW(LAST_TID-1-id, 0))\r\n#define TAGPTR_DUMMY_DESC(id) ((tagptr_t) TAGPTR_NEW(LAST_TID, id<>(offset))\r\n// TODO: make more efficient version \"MUTABLES_CAS_BIT\"\r\n// TODO: change sequence # unpacking to masking for quick comparison\r\n// note: if there is only one subfield besides seq#, then the third if-block is redundant, and you should just return false if the cas fails, since the only way the cas fails and the field being cas'd contains still old is if the sequence number has changed.\r\n#define MUTABLES_BOOL_CAS_FIELD(successBit, fldMutables, snapMutables, oldval, val, mask, offset) { \\\r\n mutables_t __v = (fldMutables); \\\r\n while (1) { \\\r\n if (UNPACK_SEQ(__v) != UNPACK_SEQ((snapMutables))) { \\\r\n (successBit) = false; \\\r\n break; \\\r\n } \\\r\n if ((successBit) = __sync_bool_compare_and_swap(&(fldMutables), \\\r\n (__v & ~(mask)) | ((oldval)<<(offset)), \\\r\n (__v & ~(mask)) | ((val)<<(offset)))) { \\\r\n break; \\\r\n } \\\r\n __v = (fldMutables); \\\r\n if (MUTABLES_UNPACK_FIELD(__v, (mask), (offset)) != (oldval)) { \\\r\n (successBit) = false; \\\r\n break; \\\r\n } \\\r\n } \\\r\n}\r\n\r\n#define MUTABLES_VAL_CAS_FIELD(failedBit, retval, fldMutables, snapMutables, oldval, val, mask, offset) { \\\r\n mutables_t __v = (fldMutables); \\\r\n while (1) { \\\r\n if (UNPACK_SEQ(__v) != UNPACK_SEQ((snapMutables))) { \\\r\n (failedBit) = true; /* version number has changed, CAS cannot occur */ \\\r\n break; \\\r\n } \\\r\n mutables_t __oldval = (__v & ~(mask)) | ((oldval)<<(offset)); \\\r\n (retval) = __sync_val_compare_and_swap(&(fldMutables), \\\r\n __oldval, \\\r\n (__v & ~(mask)) | ((val)<<(offset))); \\\r\n if ((retval) == __oldval) { /* CAS SUCCESS */ \\\r\n (retval) = MUTABLES_UNPACK_FIELD((retval), (mask), (offset)); /* return contents of subfield */ \\\r\n (failedBit) = false; \\\r\n break; \\\r\n } else { /* CAS FAILURE: should we retry? */ \\\r\n __v = (retval); /* save the value that caused our CAS to fail, in case we need to retry */ \\\r\n (retval) = MUTABLES_UNPACK_FIELD((retval), (mask), (offset)); /* return contents of subfield */ \\\r\n if ((retval) != (oldval)) { /* check if we failed because the subfield's contents do not match oldval */ \\\r\n (failedBit) = false; \\\r\n break; \\\r\n } \\\r\n /* subfield's contents DO match oldval, so we need to try again */ \\\r\n } \\\r\n } \\\r\n}\r\n\r\n// TODO: change sequence # unpacking to masking for quick comparison\r\n// note: MUTABLES_FAA_FIELD would be very similar to MUTABLES_BOOL_CAS_FIELD; i think one would simply delete the last if block and change the new val from (val)<mutables before (at least the reading portion of) the memcpy */ \\\r\n (UNPACK_SEQ(__src->mutables) == UNPACK_SEQ((tagptr))); \\\r\n})\r\n#define DESC_READ_FIELD(successBit, fldMutables, tagptr, mask, offset) ({ \\\r\n mutables_t __mutables = (fldMutables); \\\r\n successBit = (__mutables & MASK_SEQ) == ((tagptr) & MASK_SEQ); \\\r\n MUTABLES_UNPACK_FIELD(__mutables, (mask), (offset)); \\\r\n})\r\n#define DESC_NEW(descArray, macro_mutablesNew, tid) &(descArray)[(tid)]; { /* note: only the process invoking this following macro can change the sequence# */ \\\r\n SOFTWARE_BARRIER; \\\r\n mutables_t __v = (descArray)[(tid)].mutables; \\\r\n (descArray)[(tid)].mutables = macro_mutablesNew(__v); \\\r\n SOFTWARE_BARRIER; \\\r\n /*__sync_synchronize();*/ \\\r\n}\r\n#define DESC_INITIALIZED(descArray, tid) \\\r\n SOFTWARE_BARRIER; \\\r\n (descArray)[(tid)].mutables += (1<\r\n#include \r\n#include \r\n\r\n#ifndef error\r\n#define error(s) { \\\r\n std::cout<<\"ERROR: \"<\r\n#include \r\n#include \r\n#include \r\nusing namespace std;\r\n\r\ntemplate\r\nclass allocator_bump : public allocator_interface {\r\n private:\r\n const int cachelines; // # cachelines needed to store an object of type T\r\n // for bump allocation from a contiguous chunk of memory\r\n T ** mem; // mem[tid*PREFETCH_SIZE_WORDS] = pointer to current array to perform bump allocation from\r\n int * memBytes; // memBytes[tid*PREFETCH_SIZE_WORDS] = size of mem in bytes\r\n T ** current; // current[tid*PREFETCH_SIZE_WORDS] = pointer to current position in array mem\r\n vector ** toFree; // toFree[tid] = pointer to vector of bump allocation arrays to free when this allocator is destroyed\r\n\r\n T* bump_memory_next(const int tid) {\r\n T* result = current[tid*PREFETCH_SIZE_WORDS];\r\n current[tid*PREFETCH_SIZE_WORDS] = (T*) (((char*) current[tid*PREFETCH_SIZE_WORDS]) + (cachelines*BYTES_IN_CACHE_LINE));\r\n return result;\r\n }\r\n int bump_memory_bytes_remaining(const int tid) {\r\n return (((char*) mem[tid*PREFETCH_SIZE_WORDS])+memBytes[tid*PREFETCH_SIZE_WORDS]) - ((char*) current[tid*PREFETCH_SIZE_WORDS]);\r\n }\r\n bool bump_memory_full(const int tid) {\r\n return (((char*) current[tid*PREFETCH_SIZE_WORDS])+cachelines*BYTES_IN_CACHE_LINE > ((char*) mem[tid*PREFETCH_SIZE_WORDS])+memBytes[tid*PREFETCH_SIZE_WORDS]);\r\n }\r\n // call this when mem is null, or doesn't contain enough space to allocate an object\r\n void bump_memory_allocate(const int tid) {\r\n mem[tid*PREFETCH_SIZE_WORDS] = (T*) malloc(1<<24);\r\n memBytes[tid*PREFETCH_SIZE_WORDS] = 1<<24;\r\n current[tid*PREFETCH_SIZE_WORDS] = mem[tid*PREFETCH_SIZE_WORDS];\r\n toFree[tid]->push_back(mem[tid*PREFETCH_SIZE_WORDS]); // remember we allocated this to free it later\r\n#ifdef HAS_FUNCTION_aligned_alloc\r\n#else\r\n // align on cacheline boundary\r\n int mod = (int) (((long) mem[tid*PREFETCH_SIZE_WORDS]) % BYTES_IN_CACHE_LINE);\r\n if (mod > 0) {\r\n // we are ignoring the first mod bytes of mem, because if we\r\n // use them, we will not be aligning objects to cache lines.\r\n current[tid*PREFETCH_SIZE_WORDS] = (T*) (((char*) mem[tid*PREFETCH_SIZE_WORDS]) + BYTES_IN_CACHE_LINE - mod);\r\n } else {\r\n current[tid*PREFETCH_SIZE_WORDS] = mem[tid*PREFETCH_SIZE_WORDS];\r\n }\r\n#endif\r\n assert((((long) current[tid*PREFETCH_SIZE_WORDS]) % BYTES_IN_CACHE_LINE) == 0);\r\n }\r\n\r\n public:\r\n template\r\n struct rebind {\r\n typedef allocator_bump<_Tp1> other;\r\n };\r\n\r\n // reserve space for ONE object of type T\r\n T* allocate(const int tid) {\r\n // bump-allocate from a contiguous chunk of memory\r\n if (!mem[tid*PREFETCH_SIZE_WORDS] || bump_memory_full(tid)) {\r\n bump_memory_allocate(tid);\r\n MEMORY_STATS {\r\n this->debug->addAllocated(tid, memBytes[tid*PREFETCH_SIZE_WORDS] / cachelines / BYTES_IN_CACHE_LINE);\r\n VERBOSE DEBUG2 {\r\n// if ((this->debug->getAllocated(tid) % 2000) == 0) {\r\n// this->debugInterfaces->reclaim->debugPrintStatus(tid);\r\n// debugPrintStatus(tid);\r\n COUTATOMICTID(\"allocated \"<<(memBytes[tid*PREFETCH_SIZE_WORDS] / cachelines / BYTES_IN_CACHE_LINE)/*this->debug->getAllocated(tid)*/<<\" records of size \"<pool->debugPrintStatus(tid);\r\n// COUTATOMIC(endl);\r\n// }\r\n }\r\n }\r\n }\r\n return bump_memory_next(tid);\r\n }\r\n void static deallocate(const int tid, T * const p) {\r\n // no op for this allocator; memory is freed only by the destructor.\r\n // however, we have to call the destructor for the object manually...\r\n p->~T();\r\n }\r\n void deallocateAndClear(const int tid, blockbag * const bag) {\r\n // the bag is cleared, which makes it seem like we're leaking memory,\r\n // but it will be freed in the destructor as we release the huge\r\n // slabs of memory.\r\n bag->clearWithoutFreeingElements();\r\n }\r\n\r\n void debugPrintStatus(const int tid) {}\r\n\r\n void initThread(const int tid) {}\r\n \r\n allocator_bump(const int numProcesses, debugInfo * const _debug)\r\n : allocator_interface(numProcesses, _debug)\r\n , cachelines((sizeof(T)+(BYTES_IN_CACHE_LINE-1))/BYTES_IN_CACHE_LINE){\r\n VERBOSE DEBUG COUTATOMIC(\"constructor allocator_bump\"<*[numProcesses];\r\n for (int tid=0;tid();\r\n }\r\n }\r\n ~allocator_bump() {\r\n VERBOSE COUTATOMIC(\"destructor allocator_bump\"<NUM_PROCESSES;++tid) {\r\n int n = toFree[tid]->size();\r\n for (int i=0;i\r\nusing namespace std;\r\n\r\ntemplate \r\nclass allocator_interface {\r\npublic:\r\n debugInfo * const debug;\r\n \r\n const int NUM_PROCESSES;\r\n \r\n template\r\n struct rebind {\r\n typedef allocator_interface<_Tp1> other;\r\n };\r\n \r\n // allocate space for one object of type T\r\n T* allocate(const int tid);\r\n void deallocate(const int tid, T * const p);\r\n void deallocateAndClear(const int tid, blockbag * const bag);\r\n void initThread(const int tid);\r\n \r\n void debugPrintStatus(const int tid);\r\n\r\n allocator_interface(const int numProcesses, debugInfo * const _debug)\r\n : debug(_debug)\r\n , NUM_PROCESSES(numProcesses){\r\n VERBOSE DEBUG std::cout<<\"constructor allocator_interface\"<\r\n#include \r\n#include \r\nusing namespace std;\r\n\r\n//__thread long long currentAllocatedBytes = 0;\r\n//__thread long long maxAllocatedBytes = 0;\r\n\r\ntemplate\r\nclass allocator_new : public allocator_interface {\r\npublic:\r\n template\r\n struct rebind {\r\n typedef allocator_new<_Tp1> other;\r\n };\r\n \r\n // reserve space for ONE object of type T\r\n T* allocate(const int tid) {\r\n // allocate a new object\r\n MEMORY_STATS {\r\n this->debug->addAllocated(tid, 1);\r\n VERBOSE {\r\n if ((this->debug->getAllocated(tid) % 2000) == 0) {\r\n debugPrintStatus(tid);\r\n }\r\n }\r\n// currentAllocatedBytes += sizeof(T);\r\n// if (currentAllocatedBytes > maxAllocatedBytes) {\r\n// maxAllocatedBytes = currentAllocatedBytes;\r\n// }\r\n }\r\n return new T; //(T*) malloc(sizeof(T));\r\n }\r\n void deallocate(const int tid, T * const p) {\r\n // note: allocators perform the actual freeing/deleting, since\r\n // only they know how memory was allocated.\r\n // pools simply call deallocate() to request that it is freed.\r\n // allocators do not invoke pool functions.\r\n MEMORY_STATS {\r\n this->debug->addDeallocated(tid, 1);\r\n// currentAllocatedBytes -= sizeof(T);\r\n }\r\n#if !defined NO_FREE\r\n delete p;\r\n#endif\r\n }\r\n void deallocateAndClear(const int tid, blockbag * const bag) {\r\n#ifdef NO_FREE\r\n bag->clearWithoutFreeingElements();\r\n#else\r\n while (!bag->isEmpty()) {\r\n T* ptr = bag->remove();\r\n deallocate(tid, ptr);\r\n }\r\n#endif\r\n }\r\n \r\n void debugPrintStatus(const int tid) {\r\n// std::cout<debug->getAllocated(tid)<<\" objects of size \"<<(sizeof(T));\r\n// std::cout<<\" \";\r\n//// this->pool->debugPrintStatus(tid);\r\n// std::cout<(numProcesses, _debug) {\r\n VERBOSE DEBUG std::cout<<\"constructor allocator_new\"<\r\n#include \r\n#include \r\n#include \r\n#include \r\nusing namespace std;\r\n\r\n//__thread long long currentAllocatedBytes = 0;\r\n//__thread long long maxAllocatedBytes = 0;\r\n\r\ntemplate\r\nclass allocator_new_segregated : public allocator_interface {\r\nprivate:\r\n void* (*allocfn)(size_t size);\r\n void (*freefn)(void *ptr);\r\n \r\npublic:\r\n template\r\n struct rebind {\r\n typedef allocator_new_segregated<_Tp1> other;\r\n };\r\n \r\n // reserve space for ONE object of type T\r\n T* allocate(const int tid) {\r\n // allocate a new object\r\n MEMORY_STATS {\r\n this->debug->addAllocated(tid, 1);\r\n VERBOSE {\r\n if ((this->debug->getAllocated(tid) % 2000) == 0) {\r\n debugPrintStatus(tid);\r\n }\r\n }\r\n// currentAllocatedBytes += sizeof(T);\r\n// if (currentAllocatedBytes > maxAllocatedBytes) {\r\n// maxAllocatedBytes = currentAllocatedBytes;\r\n// }\r\n }\r\n return (T*) allocfn(sizeof(T));\r\n }\r\n void deallocate(const int tid, T * const p) {\r\n // note: allocators perform the actual freeing/deleting, since\r\n // only they know how memory was allocated.\r\n // pools simply call deallocate() to request that it is freed.\r\n // allocators do not invoke pool functions.\r\n MEMORY_STATS {\r\n this->debug->addDeallocated(tid, 1);\r\n// currentAllocatedBytes -= sizeof(T);\r\n }\r\n#if !defined NO_FREE\r\n p->~T(); // explicitly call destructor, since we lose automatic destructor calls when we bypass new/delete([])\r\n freefn(p);\r\n#endif\r\n }\r\n void deallocateAndClear(const int tid, blockbag * const bag) {\r\n#if defined NO_FREE\r\n bag->clearWithoutFreeingElements();\r\n#else\r\n while (!bag->isEmpty()) {\r\n T* ptr = bag->remove();\r\n deallocate(tid, ptr);\r\n }\r\n#endif\r\n }\r\n \r\n void debugPrintStatus(const int tid) {}\r\n \r\n void initThread(const int tid) {}\r\n \r\n static void* dummy_thr(void *p) { return 0; }\r\n \r\n allocator_new_segregated(const int numProcesses, debugInfo * const _debug)\r\n : allocator_interface(numProcesses, _debug) {\r\n VERBOSE DEBUG std::cout<<\"constructor allocator_new_segregated\"<\r\n#include \r\n#include \r\n#include \r\nusing namespace std;\r\n\r\n// this allocator only performs allocation once, at the beginning of the program.\r\n// define the following to specify how much memory should be allocated.\r\n#ifndef ALLOC_ONCE_MEMORY\r\n #define ALLOC_ONCE_MEMORY (1ULL<<32) /* default: 4 GB */\r\n#endif\r\n\r\n#define MIN(a, b) ((a) < (b) ? (a) : (b))\r\n\r\ntemplate\r\nclass allocator_once : public allocator_interface {\r\nprivate:\r\n const int cachelines; // # cachelines needed to store an object of type T\r\n // for bump allocation from a contiguous chunk of memory\r\n T ** mem; // mem[tid] = pointer to current array to perform bump allocation from\r\n size_t * memBytes; // memBytes[tid*PREFETCH_SIZE_WORDS] = size of mem in bytes\r\n T ** current; // current[tid*PREFETCH_SIZE_WORDS] = pointer to current position in array mem\r\n\r\n T* bump_memory_next(const int tid) {\r\n T* result = current[tid*PREFETCH_SIZE_WORDS];\r\n current[tid*PREFETCH_SIZE_WORDS] = (T*) (((char*) current[tid*PREFETCH_SIZE_WORDS]) + (cachelines*BYTES_IN_CACHE_LINE));\r\n return result;\r\n }\r\n int bump_memory_bytes_remaining(const int tid) {\r\n return (((char*) mem[tid])+memBytes[tid*PREFETCH_SIZE_WORDS]) - ((char*) current[tid*PREFETCH_SIZE_WORDS]);\r\n }\r\n bool bump_memory_full(const int tid) {\r\n return (((char*) current[tid*PREFETCH_SIZE_WORDS])+cachelines*BYTES_IN_CACHE_LINE > ((char*) mem[tid])+memBytes[tid*PREFETCH_SIZE_WORDS]);\r\n }\r\n\r\npublic:\r\n template\r\n struct rebind {\r\n typedef allocator_once<_Tp1> other;\r\n };\r\n\r\n // reserve space for ONE object of type T\r\n T* allocate(const int tid) {\r\n if (bump_memory_full(tid)) return NULL;\r\n return bump_memory_next(tid);\r\n }\r\n void static deallocate(const int tid, T * const p) {\r\n // no op for this allocator; memory is freed only by the destructor.\r\n // however, we have to call the destructor for the object manually...\r\n p->~T();\r\n }\r\n void deallocateAndClear(const int tid, blockbag * const bag) {\r\n // the bag is cleared, which makes it seem like we're leaking memory,\r\n // but it will be freed in the destructor as we release the huge\r\n // slabs of memory.\r\n bag->clearWithoutFreeingElements();\r\n }\r\n\r\n void debugPrintStatus(const int tid) {}\r\n \r\n void initThread(const int tid) {\r\n// // touch each page of memory before our trial starts\r\n// long pagesize = sysconf(_SC_PAGE_SIZE);\r\n// int last = (int) (memBytes[tid*PREFETCH_SIZE_WORDS]/pagesize);\r\n// VERBOSE COUTATOMICTID(\"touching each page... memBytes=\"<(numProcesses, _debug)\r\n , cachelines((sizeof(T)+(BYTES_IN_CACHE_LINE-1))/BYTES_IN_CACHE_LINE) {\r\n VERBOSE DEBUG COUTATOMIC(\"constructor allocator_once\"< 0) {\r\n // we are ignoring the first mod bytes of mem, because if we\r\n // use them, we will not be aligning objects to cache lines.\r\n current[tid*PREFETCH_SIZE_WORDS] = (T*) (((char*) mem[tid]) + BYTES_IN_CACHE_LINE - mod);\r\n } else {\r\n current[tid*PREFETCH_SIZE_WORDS] = mem[tid];\r\n }\r\n assert((((long) current[tid*PREFETCH_SIZE_WORDS]) % BYTES_IN_CACHE_LINE) == 0);\r\n }\r\n }\r\n ~allocator_once() {\r\n long allocated = 0;\r\n for (int tid=0;tidNUM_PROCESSES;++tid) {\r\n allocated += (((char*) current[tid*PREFETCH_SIZE_WORDS]) - ((char*) mem[tid]));\r\n }\r\n VERBOSE COUTATOMIC(\"destructor allocator_once allocated=\"<NUM_PROCESSES;++tid) {\r\n delete mem[tid];\r\n }\r\n delete[] mem;\r\n delete[] memBytes;\r\n delete[] current;\r\n }\r\n};\r\n#endif\t/* ALLOC_ONCE_H */\r\n\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/recordmgr/arraylist.h", "content": "/**\r\n * C++ record manager implementation (PODC 2015) by Trevor Brown.\r\n * \r\n * Copyright (C) 2015 Trevor Brown\r\n *\r\n */\r\n\r\n#ifndef ARRAYLIST_H\r\n#define\tARRAYLIST_H\r\n\r\n#include \r\n#include \r\n#include \r\n#include \"plaf.h\"\r\n#include \"globals.h\"\r\nusing namespace std;\r\n\r\n// this list allows multiple readers, but only ONE writer.\r\n// i don't know if it is linearizable; maybe linearize at __size.load()/store()\r\ntemplate \r\nclass AtomicArrayList {\r\nprivate:\r\n atomic_int __size;\r\n atomic_uintptr_t *data;\r\npublic:\r\n const int capacity;\r\n AtomicArrayList(const int _capacity) : capacity(_capacity) {\r\n VERBOSE DEBUG COUTATOMIC(\"constructor AtomicArrayList capacity=\"<= 0 && ix < sz);\r\n if (ix != sz-1) data[ix].store(data[sz-1].load(memory_order_relaxed), memory_order_relaxed);\r\n __size.store(sz-1, memory_order_relaxed); // note: this must be seq_cst if membars are not manually added\r\n }\r\n inline void erase(T * const obj) {\r\n int ix = getIndex(obj);\r\n if (ix != -1) erase(ix);\r\n }\r\n inline int getIndex(T * const obj) {\r\n int sz = __size.load(memory_order_relaxed); // note: this must be seq_cst if membars are not manually added\r\n for (int i=0;i\r\nclass ArrayList {\r\nprivate:\r\n int __size;\r\n T **data;\r\npublic:\r\n const int capacity;\r\n ArrayList(const int _capacity) : capacity(_capacity) {\r\n __size = 0;\r\n data = new T*[capacity];\r\n }\r\n ~ArrayList() {\r\n delete[] data;\r\n }\r\n inline T* get(const int ix) {\r\n return data[ix];\r\n }\r\n inline int size() {\r\n return __size;\r\n }\r\n inline void add(T * const obj) {\r\n assert(__size < capacity);\r\n data[__size++] = obj;\r\n }\r\n inline void erase(const int ix) {\r\n assert(ix >= 0 && ix < __size);\r\n data[ix] = data[--__size];\r\n }\r\n inline void erase(T * const obj) {\r\n int ix = getIndex(obj);\r\n if (ix != -1) erase(ix);\r\n }\r\n inline int getIndex(T * const obj) {\r\n for (int i=0;i<__size;++i) {\r\n if (data[i] == obj) return i;\r\n }\r\n return -1;\r\n }\r\n inline bool contains(T * const obj) {\r\n return (getIndex(obj) != -1);\r\n }\r\n inline void clear() {\r\n __size = 0;\r\n }\r\n inline bool isFull() {\r\n return __size == capacity;\r\n }\r\n inline bool isEmpty() {\r\n return __size == 0;\r\n }\r\n};\r\n\r\n\r\n#endif\t/* ARRAYLIST_H */\r\n\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/recordmgr/blockbag.h", "content": "/**\r\n * C++ record manager implementation (PODC 2015) by Trevor Brown.\r\n * \r\n * Copyright (C) 2015 Trevor Brown\r\n *\r\n */\r\n\r\n#ifndef BLOCKLIST_H\r\n#define\tBLOCKLIST_H\r\n\r\n#include \r\n#include \r\n#include \"blockpool.h\"\r\n#include \"plaf.h\"\r\nusing namespace std;\r\n\r\ntemplate \r\nclass blockpool;\r\n\r\ntemplate \r\nclass blockbag;\r\n\r\ntemplate \r\nclass block;\r\n\r\n#include \"lockfreeblockbag.h\"\r\n\r\n// BLOCK_SIZE must be a power of two, or else the bitwise math is invalid.\r\n#define BLOCK_SIZE (1<<8)\r\n \r\n template \r\n class block { // stack implemented as an array\r\n private:\r\n T * data[BLOCK_SIZE];\r\n int size;\r\n public:\r\n block *next;\r\n \r\n block(block * const _next) : next(_next) {\r\n size = 0;\r\n }\r\n ~block() {\r\n assert(size == 0);\r\n }\r\n \r\n bool isFull() {\r\n return size == BLOCK_SIZE;\r\n }\r\n bool isEmpty() {\r\n return size == 0;\r\n }\r\n // precondition: !isFull()\r\n void push(T * const obj) {\r\n assert(size < BLOCK_SIZE);\r\n const int sz = size;\r\n //assert(interruptible[((long) ((int *) pthread_getspecific(pthreadkey)))*PREFETCH_SIZE_WORDS] == false);\r\n data[size] = obj;\r\n SOFTWARE_BARRIER;\r\n size = sz+1;\r\n }\r\n // precondition: !isEmpty()\r\n T* pop() {\r\n assert(size > 0);\r\n const int sz = size-1;\r\n size = sz;\r\n return data[sz];\r\n }\r\n T* peek(const int ix) {\r\n assert(ix >= 0);\r\n //assert(ix < size);\r\n return data[ix];\r\n }\r\n // warning: linear time\r\n bool contains(T* const obj) {\r\n for (int i=0;i 0);\r\n if (data[size-1] == obj) {\r\n --size; // erase last pushed item\r\n return;\r\n }\r\n // the things we want to remove are probably the oldest,\r\n // so we iterate forward (head of stack = data[size-1])\r\n for (int i=0;i 0);\r\n if (ix != size-1) {\r\n data[ix] = data[size-1];\r\n }\r\n SOFTWARE_BARRIER;\r\n --size; // erase last item\r\n }\r\n }\r\n void replace(const int ix, T* const obj) {\r\n assert(ix >= 0);\r\n assert(ix < size);\r\n assert(obj);\r\n data[ix] = obj;\r\n }\r\n int computeSize() {\r\n return size;\r\n }\r\n // this function is occasionally useful if, for instance,\r\n // you use a bump allocator, which hands out objects from\r\n // a huge slab of memory.\r\n // then, in the destructor for a data structure, we can clear\r\n // a block without worrying about leaking memory,\r\n // since we will just free the whole slab at once.\r\n void clearWithoutFreeingElements() {\r\n SOFTWARE_BARRIER;\r\n size = 0;\r\n SOFTWARE_BARRIER;\r\n }\r\n };\r\n\r\n template \r\n class blockbag_iterator {\r\n private:\r\n blockbag * const bag;\r\n block * const head;\r\n block * curr;\r\n int ix;\r\n// long long reclaimCountStart;\r\n#ifdef BLOCKBAG_ITERATOR_COUNT_BLOCKS_TRAVERSED\r\n int sizeInBlocks;\r\n int blocksTraversed;\r\n#endif\r\n#ifdef BLOCKBAG_ITERATOR_COUNT_STEPS\r\n int steps;\r\n#endif\r\n public:\r\n block *getCurr() const { return curr; }\r\n int getIndex() const { return ix; }\r\n \r\n blockbag_iterator(block * const _head, blockbag * const _bag) \r\n : bag(_bag), head(_head) {\r\n#ifdef BLOCKBAG_ITERATOR_COUNT_STEPS\r\n steps = 0;\r\n#endif\r\n// reclaimCountStart = bag->getReclaimCount();\r\n// assert((reclaimCountStart % 1) == 0);\r\n// if (reclaimCountStart % 1) {\r\n// // bag is currently being reclaimed. we cannot traverse it.\r\n// curr = NULL;\r\n// ix = -1;\r\n// } else {\r\n curr = head;\r\n ix = -1;\r\n if (curr) {\r\n ix = curr->computeSize(); // \"linearize\" here\r\n (*this)++;\r\n }\r\n// }\r\n#ifdef BLOCKBAG_ITERATOR_COUNT_BLOCKS_TRAVERSED\r\n sizeInBlocks = bag->sizeInBlocks;\r\n blocksTraversed = 0;\r\n#endif\r\n }\r\n inline T* operator*() const {\r\n#ifdef BLOCKBAG_ITERATOR_COUNT_STEPS\r\n if (ix < 0) std::cout<<\"bag=\"<getReclaimCount());\r\n// if (reclaimCountStart != bag->getReclaimCount()) {\r\n// // bag is being/has been reclaimed, so we cannot iterate\r\n// return NULL;\r\n// }\r\n// /******* end consistency check for concurrent iteration *******/\r\n return curr->peek(ix);\r\n }\r\n inline blockbag_iterator& operator++(int) {\r\n#ifdef BLOCKBAG_ITERATOR_COUNT_STEPS\r\n ++steps;\r\n#endif\r\n --ix;\r\n if (ix < 0) {\r\n#ifdef BLOCKBAG_ITERATOR_COUNT_BLOCKS_TRAVERSED\r\n ++blocksTraversed;\r\n if (blocksTraversed > sizeInBlocks + 1) {\r\n std::cout<<\"ERROR: too many blocks traversed! traversed \"<next; // race condition: if reclamation happens AND curr is freed along with too many other blocks to fit in the blockpool, then this access might fault\r\n// /****** start consistency check for concurrent iteration ******/\r\n// assert(reclaimCountStart == bag->getReclaimCount());\r\n// if (reclaimCountStart != bag->getReclaimCount()) {\r\n// // bag is being/has been reclaimed, so we cannot iterate\r\n// curr = NULL;\r\n// ix = -1;\r\n// return *this;\r\n// }\r\n// /******* end consistency check for concurrent iteration *******/\r\n ix = (curr ? curr->computeSize()-1 : -1); // race condition: if reclamation happens AND curr is freed along with too many other blocks to fit in the blockpool, then this access might fault\r\n// /****** start consistency check for concurrent iteration ******/\r\n// assert(reclaimCountStart == bag->getReclaimCount());\r\n// if (reclaimCountStart != bag->getReclaimCount()) {\r\n// // bag is being/has been reclaimed, so we cannot iterate\r\n// curr = NULL;\r\n// ix = -1;\r\n// return *this;\r\n// }\r\n// /******* end consistency check for concurrent iteration *******/\r\n }\r\n return *this;\r\n }\r\n void swap(block * const otherCurr, const int otherIx) {\r\n T * const temp = otherCurr->peek(otherIx);\r\n otherCurr->replace(otherIx, curr->peek(ix));\r\n curr->replace(ix, temp);\r\n }\r\n // erases the current item\r\n void erase() {\r\n assert(curr);\r\n assert(!curr->isEmpty());\r\n bool result = bag->erase(curr, ix);\r\n if (ix >= curr->computeSize()) {\r\n (*this)++;\r\n }\r\n if (result) {\r\n (*this)++;\r\n }\r\n }\r\n };\r\n template \r\n inline bool operator==(const blockbag_iterator& a, const blockbag_iterator& b) {\r\n if (a.getCurr() != b.getCurr()) return false;\r\n if (a.getIndex() != b.getIndex()) return false;\r\n return true;\r\n }\r\n template \r\n inline bool operator!=(const blockbag_iterator& a, const blockbag_iterator& b) {\r\n return !(a == b);\r\n }\r\n \r\n // bag implemented with linked list whose nodes are blocks.\r\n // invariant: head and tail are never NULL\r\n // invariant: head is not full (computeSize() < BLOCK_SIZE)\r\n // invariant: all blocks except for the head are full\r\n // invariant: the bag is empty iff head is empty and head->next is null\r\n template \r\n class blockbag {\r\n private:\r\n int owner;\r\n volatile long long reclaimCount; // number of times this bag has been the oldest epoch bag and had its nodes reclaimed\r\n long debugFreed;\r\n public:\r\n int sizeInBlocks;\r\n private:\r\n \r\n block *head;\r\n block *tail;\r\n \r\n void validate() {\r\n // invariant: head and tail are never NULL\r\n assert(head);\r\n // invariant: head and tail are never NULL\r\n assert(tail);\r\n // invariant: head is not full (computeSize() < BLOCK_SIZE)\r\n assert(!head->isFull());\r\n // invariant: all blocks except for the head are full\r\n block *curr = head->next;\r\n while (curr) {\r\n assert(curr->isFull());\r\n curr = curr->next;\r\n }\r\n // invariant: sizeInBlocks is correct\r\n assert(sizeInBlocks == computeSizeInBlocks());\r\n }\r\n \r\n blockpool * const pool;\r\n \r\n void debugPrintBag() {\r\n std::cout<<\"(\"< * curr = head;\r\n while (curr) {\r\n std::cout<<\" \"<computeSize()<<\"[\"<<((long)curr)<<\"]\";\r\n curr = curr->next;\r\n }\r\n }\r\n int computeSizeInBlocks() {\r\n int result = 0;\r\n block *curr = head;\r\n while (curr) {\r\n ++result;\r\n curr = curr->next;\r\n }\r\n return result;\r\n }\r\n \r\n public:\r\n blockbag(const int tid, blockpool * const _pool) : pool(_pool) {\r\n// VERBOSE DEBUG std::cout<<\"constructor blockbag\"<allocateBlock(NULL);\r\n tail = head;\r\n DEBUG2 assert(computeSizeInBlocks() == sizeInBlocks);\r\n DEBUG2 assert(computeSize() == 0);\r\n DEBUG2 validate();\r\n }\r\n ~blockbag() {\r\n// VERBOSE DEBUG std::cout<<\"destructor blockbag;\";\r\n assert(isEmpty());\r\n // clear the bag AND FREE EVERY BLOCK IN IT\r\n while (head) {\r\n block * const temp = head;\r\n head = head->next;\r\n //DEBUG ++debugFreed;\r\n pool->deallocateBlock(temp);\r\n }\r\n// VERBOSE DEBUG std::cout<<\" freed \"< begin() {\r\n return blockbag_iterator(head, this);\r\n }\r\n blockbag_iterator end() {\r\n return blockbag_iterator(NULL, this);\r\n }\r\n\r\n void add(T * const obj) {\r\n DEBUG2 validate();\r\n int oldsize; DEBUG2 oldsize = computeSize();\r\n head->push(obj);\r\n if (head->isFull()) {\r\n int oldNumBlocks; DEBUG2 oldNumBlocks = computeSizeInBlocks();\r\n block *newblock = pool->allocateBlock(head);\r\n ++sizeInBlocks;\r\n //DEBUG2 std::cout<<\"(((\"<<((long)head)<<\" full. prepending \"<<((long)newblock)<<\")))\";\r\n SOFTWARE_BARRIER;\r\n head = newblock;\r\n DEBUG2 assert(oldNumBlocks + 1 == computeSizeInBlocks());\r\n DEBUG2 assert(sizeInBlocks == computeSizeInBlocks());\r\n }\r\n DEBUG2 assert(oldsize + 1 == computeSize());\r\n DEBUG2 validate();\r\n }\r\n \r\n template \r\n void add(const int tid, T * const obj, lockfreeblockbag * const sharedBag, const int thresh, Alloc * const alloc) {\r\n DEBUG2 validate();\r\n int oldsize; DEBUG2 oldsize = computeSize();\r\n head->push(obj);\r\n if (head->isFull()) {\r\n int oldNumBlocks; DEBUG2 oldNumBlocks = computeSizeInBlocks();\r\n block *newblock = pool->allocateBlock(head);\r\n ++sizeInBlocks;\r\n //DEBUG2 std::cout<<\"(((\"<<((long)head)<<\" full. prepending \"<<((long)newblock)<<\")))\";\r\n head = newblock;\r\n DEBUG2 assert(oldNumBlocks + 1 == computeSizeInBlocks());\r\n DEBUG2 assert(sizeInBlocks == computeSizeInBlocks());\r\n DEBUG2 assert(oldsize + 1 == computeSize());\r\n if (sizeInBlocks > thresh) {\r\n block *b = removeFullBlock(); // returns NULL if freeBag has < 2 full blocks\r\n assert(b);\r\n sharedBag->addBlock(b);\r\n MEMORY_STATS alloc->debug->addGiven(tid, 1);\r\n //DEBUG2 COUTATOMIC(\" thread \"<tid<<\" sharedBag(\"<<(sizeof(T)==sizeof(Node)?\"Node\":\"SCXRecord\")<<\") now contains \"<size()<<\" blocks\"<next == NULL && head->isEmpty();\r\n }\r\n // precondition: !isEmpty, !curr->isEmpty()\r\n // returns true if a subsequent invocation of curr->peek(ix) will return\r\n // an item that was previously EARLIER in iterator order, and false otherwise.\r\n bool erase(block * const curr, const int ix) {\r\n assert(!isEmpty());\r\n assert(!curr->isEmpty());\r\n DEBUG2 validate();\r\n if (head->isEmpty()) {\r\n // current block cannot be head, since head is empty\r\n assert(curr != head);\r\n \r\n // eliminate empty head block, since next block will now be non-full\r\n block * const temp = head;\r\n head = head->next;\r\n pool->deallocateBlock(temp);\r\n --sizeInBlocks;\r\n }\r\n assert(!head->isEmpty());\r\n \r\n // case 1: curr is the new head\r\n if (curr == head) {\r\n // erase from head block\r\n head->erase(ix);\r\n DEBUG2 validate();\r\n return false;\r\n \r\n // case 2: curr is not the head\r\n } else {\r\n assert(!head->isEmpty());\r\n // we use head->pop() to retrieve\r\n // some object from the head block.\r\n // then, we replace the object to be erased\r\n // with the object taken from the head block.\r\n T* obj = head->pop();\r\n curr->replace(ix, obj);\r\n DEBUG2 validate();\r\n return true;\r\n }\r\n }\r\n // precondition: !isEmpty()\r\n T* remove() {\r\n assert(!isEmpty());\r\n DEBUG2 validate();\r\n int oldsize; DEBUG2 oldsize = computeSize();\r\n T *result;\r\n if (head->isEmpty()) {\r\n result = head->next->pop();\r\n int oldNumBlocks; DEBUG2 oldNumBlocks = computeSizeInBlocks();\r\n block * const temp = head;\r\n head = head->next;\r\n pool->deallocateBlock(temp);\r\n --sizeInBlocks;\r\n DEBUG2 assert(oldNumBlocks - 1 == computeSizeInBlocks());\r\n DEBUG2 assert(sizeInBlocks == computeSizeInBlocks());\r\n DEBUG2 assert(oldsize - 1 == computeSize());\r\n DEBUG2 validate();\r\n return result;\r\n } else {\r\n result = head->pop();\r\n DEBUG2 validate();\r\n return result;\r\n }\r\n }\r\n \r\n \r\n ////////// not anymore // precondition: !isEmpty()\r\n template \r\n T* remove(const int tid, lockfreeblockbag * const sharedBag, Alloc * const alloc) {\r\n //assert(!isEmpty());\r\n DEBUG2 validate();\r\n int oldsize; DEBUG2 oldsize = computeSize();\r\n T *result;\r\n if (head->isEmpty()) {\r\n if (head->next) {\r\n result = head->next->pop();\r\n int oldNumBlocks; DEBUG2 oldNumBlocks = computeSizeInBlocks();\r\n block * const temp = head;\r\n head = head->next;\r\n pool->deallocateBlock(temp);\r\n --sizeInBlocks;\r\n DEBUG2 assert(oldNumBlocks - 1 == computeSizeInBlocks());\r\n DEBUG2 assert(sizeInBlocks == computeSizeInBlocks());\r\n DEBUG2 assert(oldsize - 1 == computeSize());\r\n// if (sizeInBlocks == 1) {\r\n// block *b = sharedBag->getBlock();\r\n// if (b) {\r\n// addFullBlock(b);\r\n// //DEBUG this->debug->addTaken(tid, 1);\r\n// //DEBUG2 COUTATOMIC(\" thread \"<tid<<\" took \"<computeSize()<<\" objects from sharedBag\"<allocate(tid));\r\n// }\r\n// /** end debug **/\r\n// }\r\n// }\r\n// assert(sizeInBlocks > 1);\r\n DEBUG2 validate();\r\n// MEMORY_STATS2 alloc->debug->addFromPool(tid, 1);\r\n return result;\r\n } else {\r\n block *b = sharedBag->getBlock();\r\n if (b) {\r\n addFullBlock(b);\r\n MEMORY_STATS alloc->debug->addTaken(tid, 1);\r\n //DEBUG2 COUTATOMIC(\" thread \"<tid<<\" took \"<computeSize()<<\" objects from sharedBag\"<allocate(tid);\r\n /** begin debug **/\r\n // allocate entire block worth of objects\r\n for (int i=0;iallocate(tid));\r\n }\r\n /** end debug **/\r\n assert(sizeInBlocks > 1);\r\n DEBUG2 validate();\r\n return remove(/*tid, sharedBag, alloc*/);\r\n }\r\n }\r\n } else {\r\n// MEMORY_STATS2 alloc->debug->addFromPool(tid, 1);\r\n result = head->pop();\r\n DEBUG2 validate();\r\n return result;\r\n }\r\n }\r\n \r\n // removes and returns a full block if the list contains\r\n // at least two full blocks. otherwise, this returns NULL;\r\n block* removeFullBlock() {\r\n DEBUG2 validate();\r\n int oldsize; DEBUG2 oldsize = computeSize();\r\n int oldNumBlocks; DEBUG2 oldNumBlocks = computeSizeInBlocks();\r\n block *second = head->next;\r\n if (second != NULL) {\r\n if (second->next != NULL) {\r\n assert(second->computeSize() == BLOCK_SIZE);\r\n head->next = second->next;\r\n second->next = NULL; // not technically necessary, but safer\r\n --sizeInBlocks;\r\n DEBUG2 assert(oldNumBlocks - 1 == computeSizeInBlocks());\r\n DEBUG2 assert(oldsize - BLOCK_SIZE == computeSize());\r\n DEBUG2 assert(sizeInBlocks == computeSizeInBlocks());\r\n DEBUG2 validate();\r\n return second;\r\n }\r\n }\r\n DEBUG2 assert(oldsize == computeSize());\r\n DEBUG2 if (sizeInBlocks != computeSizeInBlocks()) { std::cout<<\"sizeInBlocks=\"< *b) {\r\n DEBUG2 validate();\r\n assert(b->computeSize() == BLOCK_SIZE);\r\n assert(b->next == NULL);\r\n int oldsize; DEBUG2 oldsize = computeSize();\r\n int oldNumBlocks; DEBUG2 oldNumBlocks = computeSizeInBlocks();\r\n tail->next = b;\r\n tail = b;\r\n ++sizeInBlocks;\r\n DEBUG2 assert(oldNumBlocks + 1 == computeSizeInBlocks());\r\n DEBUG2 assert(oldsize + BLOCK_SIZE == computeSize());\r\n DEBUG2 assert(sizeInBlocks == computeSizeInBlocks());\r\n DEBUG2 validate();\r\n }\r\n// void appendMoveFullBlocks(blockbag * const other) {\r\n// assert(other);\r\n// assert(other->head);\r\n// DEBUG2 validate();\r\n// \r\n// // other consists of one non-full block followed by\r\n// // zero or more full blocks.\r\n// // we want this list to contain only full blocks,\r\n// // except for the head block, so we move any full blocks\r\n// // from other to this list.\r\n// // if the other blockbag has a non-empty first block, we simply\r\n// // ignore it.\r\n// // (it doesn't matter if we leave a small amount of objects in\r\n// // the other bag; they'll be appended to another list later,\r\n// // when more blocks become full.)\r\n//\r\n// // if other contains any full blocks, we append them to this list.\r\n// if (other->head->next != NULL) {\r\n// DEBUG2 assert(other->head->next->computeSize() == BLOCK_SIZE);\r\n// assert(other->head->next->isFull());\r\n// // append all but the head of the other bag to the end of this bag\r\n// sizeInBlocks += (other->getSizeInBlocks() - 1);\r\n// tail->next = other->head->next;\r\n// tail = other->tail;\r\n// assert(head && tail);\r\n// // remove all but the head of the other bag\r\n// other->head->next = NULL;\r\n// other->tail = other->head;\r\n// other->sizeInBlocks = 1;\r\n// assert(other->head && other->tail);\r\n// }\r\n// DEBUG2 other->validate();\r\n// DEBUG2 validate();\r\n// }\r\n// block * const getPredecessorBlock(block * const curr) {\r\n// block * result = head;\r\n// while (result && result != curr) {\r\n// result = result->next;\r\n// }\r\n// return result;\r\n// }\r\n void appendMoveFullBlocks(blockbag * const other, block * predecessor) {\r\n assert(other);\r\n assert(other->head);\r\n assert(predecessor);\r\n DEBUG2 validate();\r\n \r\n // other consists of one maybe-full block followed by\r\n // zero or more full blocks.\r\n // our goal is to append all blocks in the other bag\r\n // starting with predecessor->next to our own bag.\r\n if (predecessor->next != NULL) {\r\n DEBUG2 assert(predecessor->next->computeSize() == BLOCK_SIZE);\r\n assert(predecessor->next->isFull());\r\n tail->next = predecessor->next;\r\n tail = other->tail;\r\n assert(head && tail);\r\n sizeInBlocks = computeSizeInBlocks();\r\n // remove all blocks after predecessor in the other bag\r\n predecessor->next = NULL;\r\n other->tail = predecessor;\r\n other->sizeInBlocks = other->computeSizeInBlocks();\r\n assert(other->head && other->tail);\r\n }\r\n DEBUG2 other->validate();\r\n DEBUG2 validate();\r\n }\r\n void appendMoveFullBlocks(blockbag * const other) {\r\n appendMoveFullBlocks(other, other->head);\r\n }\r\n void appendMoveAll(blockbag * const other) {\r\n assert(other);\r\n DEBUG2 validate();\r\n appendMoveFullBlocks(other);\r\n while (!other->isEmpty()) {\r\n add(other->remove());\r\n }\r\n sizeInBlocks = computeSizeInBlocks();\r\n DEBUG2 validate();\r\n }\r\n int computeSize() {\r\n int result = 0;\r\n block *curr = head;\r\n while (curr) {\r\n result += curr->computeSize();\r\n curr = curr->next;\r\n }\r\n return result;\r\n }\r\n int getSizeInBlocks() {\r\n return sizeInBlocks;\r\n }\r\n // this function is occasionally useful if, for instance,\r\n // you use a bump allocator, which hands out objects from\r\n // a huge slab of memory.\r\n // then, in the destructor for a data structure, we can clear\r\n // a blockbag without worrying about leaking memory,\r\n // since we will just free the whole slab at once.\r\n void clearWithoutFreeingElements() {\r\n // free all blocks except for head.\r\n // we still have to do this, even if we don't have to\r\n // free elements of type T, since blocks are always\r\n // allocated using a blockpool, and we will leak memory\r\n // if we don't return blocks to the pool.\r\n DEBUG2 validate();\r\n block * curr = head->next;\r\n while (curr) {\r\n block * const temp = curr;\r\n curr = curr->next;\r\n temp->clearWithoutFreeingElements();\r\n this->pool->deallocateBlock(temp);\r\n }\r\n // fix up the head/tail pointers\r\n // and clear the remaining head block\r\n head->next = NULL;\r\n tail = head;\r\n head->clearWithoutFreeingElements();\r\n sizeInBlocks = 1;\r\n DEBUG2 validate();\r\n }\r\n };\r\n\r\n#endif\t/* BLOCKBAG_H */\r\n\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/recordmgr/blockpool.h", "content": "/**\r\n * C++ record manager implementation (PODC 2015) by Trevor Brown.\r\n * \r\n * Copyright (C) 2015 Trevor Brown\r\n *\r\n */\r\n\r\n#ifndef BLOCKPOOL_H\r\n#define\tBLOCKPOOL_H\r\n\r\n#include \"blockbag.h\"\r\n#include \"plaf.h\"\r\n#include \r\nusing namespace std;\r\n\r\n#define MAX_BLOCK_POOL_SIZE 32\r\n\r\n#ifndef VERBOSE\r\n#define VERBOSE if(0)\r\n#endif\r\n\r\ntemplate \r\nclass block;\r\n\r\ntemplate \r\nclass blockpool {\r\nprivate:\r\n block *pool[MAX_BLOCK_POOL_SIZE];\r\n int poolSize;\r\n\r\n long debugAllocated;\r\n long debugPoolDeallocated;\r\n long debugPoolAllocated;\r\n long debugFreed;\r\npublic:\r\n blockpool() {\r\n poolSize = 0;\r\n debugAllocated = 0;\r\n debugPoolAllocated = 0;\r\n debugPoolDeallocated = 0;\r\n debugFreed = 0;\r\n }\r\n ~blockpool() {\r\n VERBOSE DEBUG std::cout<<\"destructor blockpool;\";\r\n for (int i=0;iisEmpty());\r\n delete pool[i]; // warning: uses locks (for some allocators)\r\n }\r\n VERBOSE DEBUG std::cout<<\" blocks allocated \"<* allocateBlock(block * const next) {\r\n if (poolSize) {\r\n //DEBUG ++debugPoolAllocated;\r\n block *result = pool[--poolSize]; // pop a block off the stack\r\n *result = block(next);\r\n assert(result->next == next);\r\n assert(result->computeSize() == 0);\r\n assert(result->isEmpty());\r\n return result;\r\n } else {\r\n //DEBUG ++debugAllocated;\r\n return new block(next); // warning: uses locks (for some allocators)\r\n }\r\n }\r\n void deallocateBlock(block * const b) {\r\n assert(b->isEmpty());\r\n if (poolSize == MAX_BLOCK_POOL_SIZE) {\r\n //DEBUG ++debugFreed;\r\n// assert(poolSize < MAX_BLOCK_POOL_SIZE); // for the RQ benchmarks, we want to assert that we never free a block\r\n#ifndef NO_FREE\r\n delete b; // warning: uses locks (for some allocators)\r\n#endif\r\n } else {\r\n //DEBUG ++debugPoolDeallocated;\r\n pool[poolSize++] = b;\r\n }\r\n }\r\n};\r\n\r\n#endif\t/* BLOCKPOOL_H */\r\n\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/recordmgr/debug_info.h", "content": "/**\r\n * C++ record manager implementation (PODC 2015) by Trevor Brown.\r\n * \r\n * Copyright (C) 2015 Trevor Brown\r\n *\r\n */\r\n\r\n#ifndef DEBUG_INFO_H\r\n#define\tDEBUG_INFO_H\r\n\r\n#include \"plaf.h\"\r\n\r\nstruct _memrecl_counters {\r\n volatile char padding1[PREFETCH_SIZE_BYTES];\r\n long allocated;\r\n long deallocated;\r\n long fromPool;\r\n long toPool; // how many objects have been added to this pool\r\n long given; // how many blocks have been moved from this pool to a shared pool\r\n long taken; // how many blocks have been moved from a shared pool to this pool\r\n long retired; // how many objects have been retired\r\n volatile char padding2[PREFETCH_SIZE_BYTES];\r\n};\r\n\r\nclass debugInfo {\r\nprivate:\r\n const int NUM_PROCESSES;\r\n _memrecl_counters c[MAX_TID_POW2];\r\npublic:\r\n void clear() {\r\n for (int tid=0;tid\r\n#include \r\n#include \"plaf.h\"\r\nusing namespace std;\r\n\r\nclass debugCounter {\r\nprivate:\r\n const int NUM_PROCESSES;\r\n volatile long long * data; // data[tid*PREFETCH_SIZE_WORDS] = count for thread tid (padded to avoid false sharing)\r\npublic:\r\n void add(const int tid, const long long val) {\r\n data[tid*PREFETCH_SIZE_WORDS] += val;\r\n }\r\n void inc(const int tid) {\r\n add(tid, 1);\r\n }\r\n long long get(const int tid) {\r\n return data[tid*PREFETCH_SIZE_WORDS];\r\n }\r\n long long getTotal() {\r\n long long result = 0;\r\n for (int tid=0;tid\r\n#include \r\n\r\n#define COUTATOMIC(coutstr) /*cout<\r\n#include \r\n#include \r\n#include \"plaf.h\"\r\nusing namespace std;\r\n\r\nnamespace hashset_namespace {\r\n// note: TABLE_SIZE must be a power of two for bitwise operations below to work\r\n#define TABLE_SIZE 32\r\n#define FIRST_INDEX(key) (hash((key)) & (TABLE_SIZE-1))\r\n#define NEXT_INDEX(ix) ((ix)+1 % TABLE_SIZE)\r\n#define EMPTY_CELL 0\r\n template\r\n class hashset {\r\n private:\r\n bool cleared;\r\n K* keys[TABLE_SIZE];\r\n inline int hash(K * const key) {\r\n // MurmurHash3's integer finalizer\r\n long long k = (long long) key;\r\n k ^= k >> 33;\r\n k *= 0xff51afd7ed558ccd;\r\n k ^= k >> 33;\r\n k *= 0xc4ceb9fe1a85ec53;\r\n k ^= k >> 33;\r\n return k;\r\n }\r\n int getIndex(K * const key) {\r\n int ix;\r\n for (ix=FIRST_INDEX(key)\r\n ; keys[ix] != EMPTY_CELL && keys[ix] != key\r\n ; ix=NEXT_INDEX(ix)) {\r\n assert(ix >= 0);\r\n assert(ix < TABLE_SIZE);\r\n }\r\n assert(ix >= 0);\r\n assert(ix < TABLE_SIZE);\r\n return ix;\r\n }\r\n public:\r\n hashset() {\r\n VERBOSE DEBUG std::cout<<\"constructor hashset\"<\r\n// class AtomicHashSet {\r\n// private:\r\n// int size; // NOT ATOMICALLY ACCESSIBLE BY OTHER THREADS THAN OWNER\r\n// bool cleared; // NOT ATOMICALLY ACCESSIBLE BY OTHER THREADS THAN OWNER\r\n// atomic_uintptr_t keys[TABLE_SIZE];\r\n// inline int hash(K * const key) {\r\n// // MurmurHash3's integer finalizer\r\n// long long k = (long long) key;\r\n// k ^= k >> 33;\r\n// k *= 0xff51afd7ed558ccd;\r\n// k ^= k >> 33;\r\n// k *= 0xc4ceb9fe1a85ec53;\r\n// k ^= k >> 33;\r\n// return k;\r\n// }\r\n// int getIndex(K * const key) {\r\n// int ix;\r\n// for (ix=FIRST_INDEX(key)\r\n// ; keys[ix] != EMPTY_CELL && keys[ix] != key\r\n// ; ix=NEXT_INDEX(ix)) {\r\n// assert(ix >= 0);\r\n// assert(ix < TABLE_SIZE);\r\n// }\r\n// assert(ix >= 0);\r\n// assert(ix < TABLE_SIZE);\r\n// return ix;\r\n// }\r\n// public:\r\n// AtomicHashSet() {\r\n// VERBOSE DEBUG std::cout<<\"constructor AtomicHashSet\"<\r\n class hashset_new {\r\n private:\r\n int tableSize;\r\n K** keys;\r\n int __size;\r\n inline long hash(K * const key) {\r\n // MurmurHash3's integer finalizer\r\n long long k = (long long) key;\r\n k ^= k >> 33;\r\n k *= 0xff51afd7ed558ccd;\r\n k ^= k >> 33;\r\n k *= 0xc4ceb9fe1a85ec53;\r\n k ^= k >> 33;\r\n return k;\r\n }\r\n inline int getIndex(K * const key) {\r\n int ix = firstIndex(key);\r\n assert(ix >= 0);\r\n assert(ix < tableSize);\r\n while (true) {\r\n if (keys[ix] == EMPTY_CELL || keys[ix] == key) {\r\n return ix;\r\n }\r\n ix = nextIndex(ix);\r\n assert(ix >= 0);\r\n assert(ix < tableSize);\r\n }\r\n }\r\n inline int firstIndex(K * const key) {\r\n return (hash(key) & (tableSize-1));\r\n }\r\n inline int nextIndex(const int ix) {\r\n return ((ix+1) & (tableSize-1));\r\n }\r\n public:\r\n hashset_new(const int numberOfElements) {\r\n tableSize = 32;\r\n while (tableSize < numberOfElements*2) {\r\n tableSize *= 2;\r\n }\r\n VERBOSE DEBUG std::cout<<\"constructor hashset_new capacity=\"<\r\n#include \r\n#include \"blockbag.h\"\r\nusing namespace std;\r\n\r\n#ifndef VERBOSE\r\n#define VERBOSE if(0)\r\n#endif\r\n\r\n// lock free bag that operates on elements of the block type,\r\n// defined in blockbag.h. this class does NOT allocate or deallocate any memory.\r\n// instead, it simply chains blocks together using their next pointers.\r\n// the implementation is a stack, with push and pop at the head.\r\n// the aba problem is avoided using version numbers with a double-wide CAS.\r\n// any contention issues with using a simple stack and overhead issues with\r\n// double-wide CAS are unimportant, because operations on this bag only happen\r\n// once a process has filled up two blocks of objects and needs to hand one\r\n// off. thus, the number of operations on this class is several orders of\r\n// magnitude smaller than the number of operations on the binary search tree.\r\ntemplate \r\nclass lockfreeblockbag {\r\nprivate:\r\n struct tagged_ptr {\r\n block *ptr;\r\n long tag;\r\n };\r\n std::atomic head;\r\npublic:\r\n lockfreeblockbag() {\r\n VERBOSE DEBUG std::cout<<\"constructor lockfreeblockbag lockfree=\"< *curr = head.load(memory_order_relaxed).ptr;\r\n int debugFreed = 0;\r\n while (curr) {\r\n block * const temp = curr;\r\n curr = curr->next;\r\n //DEBUG ++debugFreed;\r\n delete temp;\r\n }\r\n VERBOSE DEBUG std::cout<<\"freed \"<* getBlock() {\r\n while (true) {\r\n tagged_ptr expHead = head.load(memory_order_relaxed);\r\n if (expHead.ptr != NULL) {\r\n if (head.compare_exchange_weak(\r\n expHead,\r\n tagged_ptr({expHead.ptr->next, expHead.tag+1}))) {\r\n block *result = expHead.ptr;\r\n result->next = NULL;\r\n return result;\r\n }\r\n } else {\r\n return NULL;\r\n }\r\n }\r\n }\r\n void addBlock(block *b) {\r\n while (true) {\r\n tagged_ptr expHead = head.load(memory_order_relaxed);\r\n b->next = expHead.ptr;\r\n if (head.compare_exchange_weak(\r\n expHead,\r\n tagged_ptr({b, expHead.tag+1}))) {\r\n return;\r\n }\r\n }\r\n }\r\n // NOT thread safe\r\n int sizeInBlocks() {\r\n int result = 0;\r\n block *curr = head.load(memory_order_relaxed).ptr;\r\n while (curr) {\r\n ++result;\r\n curr = curr->next;\r\n }\r\n return result;\r\n }\r\n // thread safe, but concurrent operations are very likely to starve it\r\n long long size() {\r\n while (1) {\r\n long long result = 0;\r\n block *originalHead = head.load(memory_order_relaxed).ptr;\r\n block *curr = originalHead;\r\n while (curr) {\r\n result += curr->computeSize();\r\n curr = curr->next;\r\n }\r\n if (head.load(memory_order_relaxed).ptr == originalHead) {\r\n return result;\r\n }\r\n }\r\n }\r\n};\r\n\r\n#endif\t/* LOCKFREESTACK_H */\r\n\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/recordmgr/pool_interface.h", "content": "/**\r\n * C++ record manager implementation (PODC 2015) by Trevor Brown.\r\n * \r\n * Copyright (C) 2015 Trevor Brown\r\n *\r\n */\r\n\r\n#ifndef POOL_INTERFACE_H\r\n#define\tPOOL_INTERFACE_H\r\n\r\n#include \r\n#include \"allocator_interface.h\"\r\n#include \"debug_info.h\"\r\n#include \"blockpool.h\"\r\n#include \"blockbag.h\"\r\nusing namespace std;\r\n\r\ntemplate >\r\nclass pool_interface {\r\npublic:\r\n debugInfo * const debug;\r\n \r\n const int NUM_PROCESSES;\r\n blockpool **blockpools; // allocated (or not) and freed by descendants\r\n Alloc *alloc;\r\n\r\n template\r\n struct rebind {\r\n typedef pool_interface<_Tp1, Alloc> other;\r\n };\r\n template\r\n struct rebind2 {\r\n typedef pool_interface<_Tp1, _Tp2> other;\r\n };\r\n \r\n string getSizeString() { return \"\"; }\r\n// long long getSizeInNodes() { return 0; }\r\n /**\r\n * if the pool contains any object, then remove one from the pool\r\n * and return a pointer to it. otherwise, return NULL.\r\n */\r\n inline T* get(const int tid);\r\n inline void add(const int tid, T* ptr);\r\n inline void addMoveFullBlocks(const int tid, blockbag *bag);\r\n inline void addMoveAll(const int tid, blockbag *bag);\r\n inline int computeSize(const int tid);\r\n \r\n void debugPrintStatus(const int tid);\r\n \r\n pool_interface(const int numProcesses, Alloc * const _alloc, debugInfo * const _debug)\r\n : debug(_debug) \r\n , NUM_PROCESSES(numProcesses)\r\n , alloc(_alloc){\r\n VERBOSE DEBUG std::cout<<\"constructor pool_interface\"<blockpools = new blockpool*[numProcesses];\r\n for (int tid=0;tidblockpools[tid] = new blockpool();\r\n }\r\n }\r\n ~pool_interface() {\r\n VERBOSE DEBUG std::cout<<\"destructor pool_interface\"<NUM_PROCESSES;++tid) {\r\n delete this->blockpools[tid];\r\n }\r\n delete[] this->blockpools;\r\n }\r\n};\r\n\r\n#endif\r\n\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/recordmgr/pool_none.h", "content": "/**\r\n * C++ record manager implementation (PODC 2015) by Trevor Brown.\r\n * \r\n * Copyright (C) 2015 Trevor Brown\r\n *\r\n */\r\n\r\n#ifndef POOL_NOOP_H\r\n#define\tPOOL_NOOP_H\r\n\r\n#include \r\n#include \r\n#include \"blockbag.h\"\r\n#include \"blockpool.h\"\r\n#include \"pool_interface.h\"\r\n#include \"plaf.h\"\r\nusing namespace std;\r\n\r\ntemplate >\r\nclass pool_none : public pool_interface {\r\npublic:\r\n template\r\n struct rebind {\r\n typedef pool_none<_Tp1, Alloc> other;\r\n };\r\n template\r\n struct rebind2 {\r\n typedef pool_none<_Tp1, _Tp2> other;\r\n };\r\n \r\n string getSizeString() { return \"no pool\"; }\r\n /**\r\n * if the freebag contains any object, then remove one from the freebag\r\n * and return a pointer to it.\r\n * if not, then retrieve a new object from Alloc\r\n */\r\n inline T* get(const int tid) {\r\n MEMORY_STATS2 this->alloc->debug->addFromPool(tid, 1);\r\n return this->alloc->allocate(tid);\r\n }\r\n inline void add(const int tid, T* ptr) {\r\n this->alloc->deallocate(tid, ptr);\r\n }\r\n inline void addMoveFullBlocks(const int tid, blockbag *bag, block * const predecessor) {\r\n bag->clearWithoutFreeingElements();\r\n // note: this will leak memory, but i believe it is only used by debraplus (which really should use a pool)\r\n }\r\n inline void addMoveFullBlocks(const int tid, blockbag *bag) {\r\n this->alloc->deallocateAndClear(tid, bag);\r\n// T* ptr;\r\n// while (ptr = bag->remove()) {\r\n// add(tid, ptr);\r\n// }\r\n }\r\n inline void addMoveAll(const int tid, blockbag *bag) {\r\n this->alloc->deallocateAndClear(tid, bag);\r\n// T* ptr;\r\n// while (ptr = bag->remove()) {\r\n// add(tid, ptr);\r\n// }\r\n }\r\n inline int computeSize(const int tid) {\r\n return 0;\r\n }\r\n \r\n void debugPrintStatus(const int tid) {\r\n\r\n }\r\n \r\n pool_none(const int numProcesses, Alloc * const _alloc, debugInfo * const _debug)\r\n : pool_interface(numProcesses, _alloc, _debug) {\r\n VERBOSE DEBUG std::cout<<\"constructor pool_none\"<\r\n#include \r\n#include \r\n#include \"blockbag.h\"\r\n#include \"blockpool.h\"\r\n#include \"pool_interface.h\"\r\n#include \"plaf.h\"\r\n#include \"globals.h\"\r\nusing namespace std;\r\n\r\n#define POOL_THRESHOLD_IN_BLOCKS 10\r\n\r\ntemplate >\r\nclass pool_perthread_and_shared : public pool_interface {\r\nprivate:\r\n lockfreeblockbag *sharedBag; // shared bag that we offload blocks on when we have too many in our freeBag\r\n blockbag **freeBag; // freeBag[tid] = bag of objects of type T that are ready to be reused by the thread with id tid\r\n\r\n // note: only does something if freeBag contains at least two full blocks\r\n inline bool tryGiveFreeObjects(const int tid) {\r\n if (freeBag[tid]->getSizeInBlocks() >= POOL_THRESHOLD_IN_BLOCKS) {\r\n block *b = freeBag[tid]->removeFullBlock(); // returns NULL if freeBag has < 2 full blocks\r\n assert(b);\r\n// if (b) {\r\n sharedBag->addBlock(b);\r\n MEMORY_STATS this->debug->addGiven(tid, 1);\r\n //DEBUG2 COUTATOMIC(\" thread \"<tid<<\" sharedBag(\"<<(sizeof(T)==sizeof(Node)?\"Node\":\"SCXRecord\")<<\") now contains \"<size()<<\" blocks\"< *b = sharedBag->getBlock();\r\n// if (b) {\r\n// freeBag[tid]->addFullBlock(b);\r\n// DEBUG this->debug->addTaken(tid, 1);\r\n// //DEBUG2 COUTATOMIC(\" thread \"<tid<<\" took \"<computeSize()<<\" objects from sharedBag\"<\r\n struct rebind {\r\n typedef pool_perthread_and_shared<_Tp1, Alloc> other;\r\n };\r\n template\r\n struct rebind2 {\r\n typedef pool_perthread_and_shared<_Tp1, _Tp2> other;\r\n };\r\n \r\n// long long getSizeInNodes() {\r\n// long long sum = 0;\r\n// for (int tid=0;tidNUM_PROCESSES;++tid) {\r\n// sum += freeBag[tid]->computeSize();\r\n// }\r\n//// sum += sharedBag->sizeInBlocks() * BLOCK_SIZE;\r\n// return sum;\r\n// }\r\n string getSizeString() {\r\n stringstream ss;\r\n long long insharedbag = sharedBag->size();\r\n long long infreebags = 0;\r\n for (int tid=0;tidNUM_PROCESSES;++tid) {\r\n infreebags += freeBag[tid]->computeSize();\r\n }\r\n ss<alloc->debug->addFromPool(tid, 1);\r\n return freeBag[tid]->template remove(tid, sharedBag, this->alloc);\r\n }\r\n inline void add(const int tid, T* ptr) {\r\n MEMORY_STATS2 this->debug->addToPool(tid, 1);\r\n freeBag[tid]->add(tid, ptr, sharedBag, POOL_THRESHOLD_IN_BLOCKS, this->alloc);\r\n }\r\n inline void addMoveFullBlocks(const int tid, blockbag *bag, block * const predecessor) {\r\n // WARNING: THE FOLLOWING DEBUG COMPUTATION GETS THE WRONG NUMBER OF BLOCKS.\r\n MEMORY_STATS2 this->debug->addToPool(tid, (bag->getSizeInBlocks()-1)*BLOCK_SIZE);\r\n freeBag[tid]->appendMoveFullBlocks(bag, predecessor);\r\n while (tryGiveFreeObjects(tid)) {}\r\n }\r\n inline void addMoveFullBlocks(const int tid, blockbag *bag) {\r\n // WARNING: THE FOLLOWING DEBUG COMPUTATION GETS THE WRONG NUMBER OF BLOCKS.\r\n MEMORY_STATS2 this->debug->addToPool(tid, (bag->getSizeInBlocks()-1)*BLOCK_SIZE);\r\n freeBag[tid]->appendMoveFullBlocks(bag);\r\n while (tryGiveFreeObjects(tid)) {}\r\n }\r\n inline void addMoveAll(const int tid, blockbag *bag) {\r\n MEMORY_STATS2 this->debug->addToPool(tid, bag->computeSize());\r\n freeBag[tid]->appendMoveAll(bag);\r\n while (tryGiveFreeObjects(tid)) {}\r\n }\r\n inline int computeSize(const int tid) {\r\n return freeBag[tid]->computeSize();\r\n }\r\n \r\n void debugPrintStatus(const int tid) {\r\n// long free = computeSize(tid);\r\n// long share = sharedBag->sizeInBlocks();\r\n// COUTATOMIC(\"free=\"<(numProcesses, _alloc, _debug) {\r\n VERBOSE DEBUG COUTATOMIC(\"constructor pool_perthread_and_shared\"<*[numProcesses];\r\n for (int tid=0;tid(tid, this->blockpools[tid]);\r\n }\r\n sharedBag = new lockfreeblockbag();\r\n }\r\n ~pool_perthread_and_shared() {\r\n VERBOSE DEBUG COUTATOMIC(\"destructor pool_perthread_and_shared\"< *fullBlock;\r\n while ((fullBlock = sharedBag->getBlock()) != NULL) {\r\n while (!fullBlock->isEmpty()) {\r\n T * const ptr = fullBlock->pop();\r\n this->alloc->deallocate(dummyTid, ptr);\r\n }\r\n this->blockpools[dummyTid]->deallocateBlock(fullBlock);\r\n }\r\n // clean up free bags\r\n for (int tid=0;tidNUM_PROCESSES;++tid) {\r\n this->alloc->deallocateAndClear(tid, freeBag[tid]);\r\n delete freeBag[tid];\r\n }\r\n delete[] freeBag;\r\n delete sharedBag;\r\n }\r\n};\r\n\r\n#endif\r\n\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/recordmgr/reclaimer_debra.h", "content": "/**\r\n * C++ record manager implementation (PODC 2015) by Trevor Brown.\r\n * \r\n * Copyright (C) 2015 Trevor Brown\r\n *\r\n */\r\n\r\n#ifndef RECLAIM_EPOCH_H\r\n#define\tRECLAIM_EPOCH_H\r\n\r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \"blockbag.h\"\r\n#include \"plaf.h\"\r\n#include \"allocator_interface.h\"\r\n#include \"reclaimer_interface.h\"\r\nusing namespace std;\r\n\r\n\r\n\r\ntemplate >\r\nclass reclaimer_debra : public reclaimer_interface {\r\nprotected:\r\n#define EPOCH_INCREMENT 2\r\n#define BITS_EPOCH(ann) ((ann)&~(EPOCH_INCREMENT-1))\r\n#define QUIESCENT(ann) ((ann)&1)\r\n#define GET_WITH_QUIESCENT(ann) ((ann)|1)\r\n\r\n#ifdef RAPID_RECLAMATION\r\n#define MIN_OPS_BEFORE_READ 1\r\n//#define MIN_OPS_BEFORE_CAS_EPOCH 1\r\n#else\r\n#define MIN_OPS_BEFORE_READ 20\r\n//#define MIN_OPS_BEFORE_CAS_EPOCH 100\r\n#endif\r\n \r\n#define NUMBER_OF_EPOCH_BAGS 9\r\n#define NUMBER_OF_ALWAYS_EMPTY_EPOCH_BAGS 3\r\n \r\n // for epoch based reclamation\r\n volatile long epoch;\r\n atomic_long *announcedEpoch; // announcedEpoch[tid*PREFETCH_SIZE_WORDS] // todo: figure out if volatile here would help processes notice changes more quickly.\r\n long *checked; // checked[tid*PREFETCH_SIZE_WORDS] = how far we've come in checking the announced epochs of other threads\r\n blockbag **epochbags; // epochbags[NUMBER_OF_EPOCH_BAGS*tid+0..NUMBER_OF_EPOCH_BAGS*tid+(NUMBER_OF_EPOCH_BAGS-1)] are epoch bags for thread tid.\r\n blockbag **currentBag; // pointer to current epoch bag for each process\r\n long *index; // index of currentBag in epochbags for each process\r\n // note: oldest bag is number (index+1)%NUMBER_OF_EPOCH_BAGS\r\n long *opsSinceRead;\r\n \r\npublic:\r\n template\r\n struct rebind {\r\n typedef reclaimer_debra<_Tp1, Pool> other;\r\n };\r\n template\r\n struct rebind2 {\r\n typedef reclaimer_debra<_Tp1, _Tp2> other;\r\n };\r\n \r\n// inline int getOldestBlockbagIndexOffset(const int tid) {\r\n// long long min_val = LLONG_MAX;\r\n// int min_i = -1;\r\n// for (int i=0;igetReclaimCount();\r\n// if (reclaimCount % 1) { // bag's contents are currently being freed\r\n// return i;\r\n// }\r\n// if (reclaimCount < min_val) {\r\n// min_val = reclaimCount;\r\n// min_i = i;\r\n// }\r\n// }\r\n// return min_i;\r\n// }\r\n// \r\n// inline set_of_bags getBlockbags() { // blockbag_iterator ** const output) {\r\n//// int cnt=0;\r\n//// for (int tid=0;tidNUM_PROCESSES*NUMBER_OF_EPOCH_BAGS};\r\n// }\r\n// \r\n// inline void getOldestTwoBlockbags(const int tid, blockbag ** oldest, blockbag ** secondOldest) {\r\n// long long min_val = LLONG_MAX;\r\n// int min_i = -1;\r\n// for (int i=0;igetReclaimCount();\r\n// if (reclaimCount % 1) { // bag's contents are currently being freed\r\n// min_i = i;\r\n// break;\r\n// }\r\n// if (reclaimCount < min_val) {\r\n// min_val = reclaimCount;\r\n// min_i = i;\r\n// }\r\n// }\r\n// if (min_i == -1) {\r\n// *oldest = *secondOldest = NULL;\r\n// } else {\r\n// *oldest = epochbags[tid*NUMBER_OF_EPOCH_BAGS + min_i];\r\n// *secondOldest = epochbags[tid*NUMBER_OF_EPOCH_BAGS + ((min_i+1)%NUMBER_OF_EPOCH_BAGS)];\r\n// }\r\n// }\r\n \r\n inline void getSafeBlockbags(const int tid, blockbag ** bags) {\r\n SOFTWARE_BARRIER;\r\n int ix = index[tid*PREFETCH_SIZE_WORDS];\r\n bags[0] = epochbags[tid*NUMBER_OF_EPOCH_BAGS+ix];\r\n bags[1] = epochbags[tid*NUMBER_OF_EPOCH_BAGS+((ix+NUMBER_OF_EPOCH_BAGS-1)%NUMBER_OF_EPOCH_BAGS)];\r\n bags[2] = epochbags[tid*NUMBER_OF_EPOCH_BAGS+((ix+NUMBER_OF_EPOCH_BAGS-2)%NUMBER_OF_EPOCH_BAGS)];\r\n bags[3] = NULL;\r\n SOFTWARE_BARRIER;\r\n \r\n// SOFTWARE_BARRIER;\r\n// // find first dangerous blockbag\r\n// long long min_val = LLONG_MAX;\r\n// int min_i = -1;\r\n// for (int i=0;igetReclaimCount();\r\n// if (reclaimCount % 1) { // bag's contents are currently being freed\r\n// min_i = i;\r\n// break;\r\n// }\r\n// if (reclaimCount < min_val) {\r\n// min_val = reclaimCount;\r\n// min_i = i;\r\n// }\r\n// }\r\n// assert(min_i != -1);\r\n// min_i = (min_i + NUMBER_OF_ALWAYS_EMPTY_EPOCH_BAGS) % NUMBER_OF_EPOCH_BAGS;\r\n// \r\n// // process might free from bag at offset min_i, or the next one.\r\n// // the others are safe.\r\n// int i;\r\n// for (i=0;iNUM_PROCESSES;++tid) {\r\n for (int j=0;jcomputeSize();\r\n }\r\n }\r\n return sum;\r\n }\r\n string getSizeString() {\r\n stringstream ss;\r\n ss< * const freeable = epochbags[NUMBER_OF_EPOCH_BAGS*tid + ((nextIndex+NUMBER_OF_ALWAYS_EMPTY_EPOCH_BAGS) % NUMBER_OF_EPOCH_BAGS)];\r\n this->pool->addMoveFullBlocks(tid, freeable); // moves any full blocks (may leave a non-full block behind)\r\n SOFTWARE_BARRIER;\r\n index[tid*PREFETCH_SIZE_WORDS] = nextIndex;\r\n currentBag[tid*PREFETCH_SIZE_WORDS] = epochbags[NUMBER_OF_EPOCH_BAGS*tid + nextIndex];\r\n }\r\n\r\n // objects reclaimed by this epoch manager.\r\n // returns true if the call rotated the epoch bags for thread tid\r\n // (and reclaimed any objects retired two epochs ago).\r\n // otherwise, the call returns false.\r\n inline bool leaveQuiescentState(const int tid, void * const * const reclaimers, const int numReclaimers) {\r\n SOFTWARE_BARRIER; // prevent any bookkeeping from being moved after this point by the compiler.\r\n bool result = false;\r\n\r\n // ver 1\r\n long readEpoch = epoch;\r\n const long ann = announcedEpoch[tid*PREFETCH_SIZE_WORDS].load(memory_order_relaxed);\r\n// // debug ver2\r\n// const long ann = announcedEpoch[tid*PREFETCH_SIZE_WORDS].load(memory_order_relaxed);\r\n// ++opsSinceRead[tid*PREFETCH_SIZE_WORDS];\r\n// long readEpoch = ((opsSinceRead[tid*PREFETCH_SIZE_WORDS] % MIN_OPS_BEFORE_READ) == 0) ? epoch : BITS_EPOCH(ann);\r\n\r\n // if our announced epoch is different from the current epoch\r\n if (readEpoch != BITS_EPOCH(ann)) {\r\n // announce the new epoch, and rotate the epoch bags and\r\n // reclaim any objects retired two epochs ago.\r\n checked[tid*PREFETCH_SIZE_WORDS] = 0;\r\n //rotateEpochBags(tid);\r\n for (int i=0;i * const) reclaimers[i])->rotateEpochBags(tid);\r\n }\r\n result = true;\r\n }\r\n // note: readEpoch, when written to announcedEpoch[tid],\r\n // will set the state to non-quiescent and non-neutralized\r\n\r\n // incrementally scan the announced epochs of all threads\r\n int otherTid = checked[tid*PREFETCH_SIZE_WORDS];\r\n if ((++opsSinceRead[tid*PREFETCH_SIZE_WORDS] % MIN_OPS_BEFORE_READ) == 0) {\r\n long otherAnnounce = announcedEpoch[otherTid*PREFETCH_SIZE_WORDS].load(memory_order_relaxed);\r\n if (BITS_EPOCH(otherAnnounce) == readEpoch\r\n || QUIESCENT(otherAnnounce)) {\r\n const int c = ++checked[tid*PREFETCH_SIZE_WORDS];\r\n if (c >= this->NUM_PROCESSES /*&& c > MIN_OPS_BEFORE_CAS_EPOCH*/) {\r\n __sync_bool_compare_and_swap(&epoch, readEpoch, readEpoch+EPOCH_INCREMENT);\r\n }\r\n }\r\n }\r\n SOFTWARE_BARRIER;\r\n if (readEpoch != ann) {\r\n announcedEpoch[tid*PREFETCH_SIZE_WORDS].store(readEpoch, memory_order_relaxed);\r\n }\r\n return result;\r\n }\r\n \r\n inline void enterQuiescentState(const int tid) {\r\n const long ann = announcedEpoch[tid*PREFETCH_SIZE_WORDS].load(memory_order_relaxed);\r\n announcedEpoch[tid*PREFETCH_SIZE_WORDS].store(GET_WITH_QUIESCENT(ann), memory_order_relaxed);\r\n }\r\n \r\n // for all schemes except reference counting\r\n inline void retire(const int tid, T* p) {\r\n currentBag[tid*PREFETCH_SIZE_WORDS]->add(p);\r\n DEBUG2 this->debug->addRetired(tid, 1);\r\n }\r\n \r\n inline void unretireLast(const int tid) {\r\n assert(false); // we do not use this, since it makes it harder to reason about iteration over blockbags when they shrink (aside from when their contents are being reclaimed, and we can determine this is the case by inspecting bag->getReclaimCount()...)\r\n currentBag[tid*PREFETCH_SIZE_WORDS]->remove();\r\n }\r\n\r\n void debugPrintStatus(const int tid) {\r\n// assert(tid >= 0);\r\n// assert(tid < this->NUM_PROCESSES);\r\n if (tid == 0) {\r\n std::cout<<\"global epoch counter=\"<computeSize();\r\n// }\r\n// std::cout<<\")\"< * const _recoveryMgr = NULL)\r\n : reclaimer_interface(numProcesses, _pool, _debug, _recoveryMgr) {\r\n VERBOSE std::cout<<\"constructor reclaimer_debra helping=\"<shouldHelp()<*[NUMBER_OF_EPOCH_BAGS*numProcesses];\r\n currentBag = new blockbag*[numProcesses*PREFETCH_SIZE_WORDS];\r\n index = new long[numProcesses*PREFETCH_SIZE_WORDS];\r\n opsSinceRead = new long[numProcesses*PREFETCH_SIZE_WORDS];\r\n announcedEpoch = new atomic_long[numProcesses*PREFETCH_SIZE_WORDS];\r\n checked = new long[numProcesses*PREFETCH_SIZE_WORDS];\r\n for (int tid=0;tid(tid, this->pool->blockpools[tid]);\r\n }\r\n currentBag[tid*PREFETCH_SIZE_WORDS] = epochbags[NUMBER_OF_EPOCH_BAGS*tid];\r\n index[tid*PREFETCH_SIZE_WORDS] = 0;\r\n opsSinceRead[tid*PREFETCH_SIZE_WORDS] = 0;\r\n announcedEpoch[tid*PREFETCH_SIZE_WORDS].store(GET_WITH_QUIESCENT(0), memory_order_relaxed);\r\n checked[tid*PREFETCH_SIZE_WORDS] = 0;\r\n }\r\n }\r\n ~reclaimer_debra() {\r\n VERBOSE DEBUG std::cout<<\"destructor reclaimer_debra\"<NUM_PROCESSES;++tid) {\r\n // move contents of all bags into pool\r\n for (int i=0;icomputeSize()<<\" objects from epoch bag of tid=\"<pool->addMoveAll(tid, epochbags[NUMBER_OF_EPOCH_BAGS*tid+i]);\r\n delete epochbags[NUMBER_OF_EPOCH_BAGS*tid+i];\r\n }\r\n }\r\n delete[] epochbags;\r\n delete[] index;\r\n delete[] opsSinceRead;\r\n delete[] currentBag;\r\n delete[] announcedEpoch;\r\n delete[] checked;\r\n }\r\n\r\n};\r\n\r\n#endif\r\n\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/recordmgr/reclaimer_debraplus.h", "content": "/**\r\n * C++ record manager implementation (PODC 2015) by Trevor Brown.\r\n * \r\n * Copyright (C) 2015 Trevor Brown\r\n *\r\n */\r\n\r\n#ifndef RECLAIM_EPOCH_CRASHRECOV_H\r\n#define\tRECLAIM_EPOCH_CRASHRECOV_H\r\n\r\n#include \r\n#include \r\n#include \"plaf.h\"\r\n#include \"globals.h\"\r\n#include \"blockbag.h\"\r\n#include \"allocator_interface.h\"\r\n#include \"reclaimer_interface.h\"\r\n#include \"arraylist.h\"\r\n#include \"hashtable.h\"\r\n#include \"record_manager_single_type.h\"\r\nusing namespace std;\r\nusing namespace hashset_namespace;\r\n\r\ntemplate >\r\nclass reclaimer_debraplus : public reclaimer_interface {\r\nprivate:\r\n#define EPOCH_INCREMENT 2\r\n#define BITS_EPOCH(ann) ((ann)&~(EPOCH_INCREMENT-1))\r\n#define QUIESCENT(ann) ((ann)&1)\r\n#define GET_WITH_QUIESCENT(ann) ((ann)|1)\r\n// the following threshold allows a process to accumulate about 768 objects in each epoch bag\r\n// (3*BLOCK_SIZE=768, but there are other things that inflate bag size slightly, such as\r\n// the fact that a thread can do n operations before it successfully neutralizes each thread\r\n// and can advance the epoch.)\r\n#define NEUTRALIZE_THRESHOLD_IN_BLOCKS 4\r\n// maximum number of objects that can be simultaneously protected by calls to qProtect()\r\n#define MAX_PROTECT_EVEN_IF_QUIESCENT 7\r\n\r\n#define MINIMUM_OPERATIONS_BEFORE_NEW_EPOCH_CR 100\r\n#define NUMBER_OF_EPOCH_BAGS_CR 3\r\n // for epoch based reclamation\r\n volatile long epoch;\r\n atomic_long *announcedEpoch; // announcedEpoch[tid*PREFETCH_SIZE_WORDS] = bits 1..end contain the last epoch seen by thread tid, and bit 0 indicates quiescence\r\n long *checked; // checked[tid*PREFETCH_SIZE_WORDS] = how far we've come in checking the announced epochs of other threads\r\n blockbag **epochbags; // epochbags[NUMBER_OF_EPOCH_BAGS*tid+0..NUMBER_OF_EPOCH_BAGS*tid+(NUMBER_OF_EPOCH_BAGS-1)] are epoch bags for thread tid.\r\n blockbag **currentBag; // pointer to current epoch bag for each process\r\n long *index; // index of currentBag in epochbags for each process\r\n // note: oldest bag is number (index+1)%NUMBER_OF_EPOCH_BAGS_CR\r\n\r\n // for hazard pointer component of this scheme;\r\n // each thread has a single hazard pointer that it uses to prevent\r\n // other threads from reclaiming its current scx record before it can\r\n // clean up after itself.\r\n AtomicArrayList **announce; // announce[tid] = pointer to set of hazard pointers for thread tid\r\n hashset_new **comparing; // comparing[tid] = set of announced hazard pointers for ALL threads, as collected by thread tid during it's last retire(tid, ...) call\r\n\r\n // number of blocks retired[tid] must contain before it is guaranteed to\r\n // contain at least 5*numProcesses*MAX_PROTECT_EVEN_IF_QUIESCENT items...\r\n // why 5*numProcesses*MAX_PROTECT_EVEN_IF_QUIESCENT items?\r\n // to get amortized constant scanning time per object,\r\n // the number of elements that retired[tid] must contain\r\n // before we scan hazard pointers to determine\r\n // which elements of retired[tid] can be deallocated\r\n // must be nk+Omega(nk), where\r\n // n = number of threads and\r\n // k = max number of hazard pointers a thread can hold at once\r\n // in this context, k=MAX_PROTECT_EVEN_IF_QUIESCENT, since a thread only obtains\r\n // a hazard pointer to the scx record it has most recently created, and\r\n // the nodes it points to. so, we just need some constant times\r\n // numProcesses*MAX_PROTECT_EVEN_IF_QUIESCENT.\r\n static const int scanThreshold = 4;\r\n\r\n sigset_t neutralizeSignalSet;\r\n \r\n inline bool neutralizeOther(const int tid, const int otherTid, const long currentEpoch, const long announceOther) {\r\n#ifdef SEND_CRASH_RECOVERY_SIGNALS\r\n assert(isQuiescent(tid));\r\n assert(otherTid != tid);\r\n // if the epoch bag is too full, then we suspect otherTid has crashed...\r\n if (epochbags[NUMBER_OF_EPOCH_BAGS_CR*tid+index[tid*PREFETCH_SIZE_WORDS]]->getSizeInBlocks() >= NEUTRALIZE_THRESHOLD_IN_BLOCKS) {\r\n \r\n // neutralize otherTid by sending him a signal to make him\r\n // change what his next step will be, and force him to\r\n // throw away all pointers into the data structure, and\r\n // leaveQstate again before re-acquiring any pointers into\r\n // the data structure. this lets us reclaim memory without\r\n // waiting for him to progress.\r\n pthread_t otherPthread = this->recoveryMgr->getPthread(otherTid);\r\n int error = 0;\r\n// COUTATOMICTID(\"sending signal to tid \"<recoveryMgr->neutralizeSignal)) {\r\n // should never happen\r\n for (int i=0;i<20;++i) COUTATOMICTID(\"######################################################\"<recoveryMgr->neutralizeSignal<<\")\"<\r\n struct rebind {\r\n typedef reclaimer_debraplus<_Tp1, Pool> other;\r\n };\r\n template\r\n struct rebind2 {\r\n typedef reclaimer_debraplus<_Tp1, _Tp2> other;\r\n };\r\n \r\n inline static bool quiescenceIsPerRecordType() { return false; }\r\n inline static bool supportsCrashRecovery() { return true; }\r\n inline bool isQuiescent(const int tid) {\r\n //COUTATOMICTID(\"IS QUIESCENT EXECUTED\"<contains(obj); // this is inefficient, but should only happen when recovering from being neutralized...\r\n }\r\n inline bool qProtect(const int tid, T * const obj, CallbackType notRetiredCallback, CallbackArg callbackArg, bool memoryBarrier = true) {\r\n TRACE COUTATOMICTID(\"reclaimer_debraplus::protectObjectEvenIfQuiescent(tid=\"<size();\r\n DEBUG assert(__size < MAX_PROTECT_EVEN_IF_QUIESCENT);\r\n announce[tid]->add(obj);\r\n assert(announce[tid]->contains(obj));\r\n DEBUG assert(announce[tid]->size() == __size+1);\r\n // if callbackArg = NULL, we assume notRetiredCallback is a noop.\r\n if (memoryBarrier) __sync_synchronize(); // prevent retired from being read before we set a hazard pointer to obj, and prevent any future reads of fields of obj from being moved before we announce obj.\r\n if (notRetiredCallback(callbackArg)) {\r\n TRACE COUTATOMICTID(\"notRetiredCallback returns true\"<erase(obj); // note: this is inefficient, but it should never happen with regular use.\r\n DEBUG assert(__size == announce[tid]->size());\r\n return false;\r\n }\r\n }\r\n inline void qUnprotectAll(const int tid) {\r\n TRACE COUTATOMICTID(\"reclaimer_debraplus::unprotectAllObjectsEvenIfQuiescent(tid=\"<clear();\r\n assert(announce[tid]->size() == 0);\r\n }\r\n \r\n // rotate the epoch bags and reclaim any objects retired two epochs ago.\r\n inline void rotateEpochBags(const int tid) {\r\n assert(isQuiescent(tid));\r\n // we rotate lists in constant time, and scan hazard pointers\r\n // when the blockbag from two epochs ago is larger than scanThreshold\r\n // (using an iterator with erase functionality).\r\n // maybe in the future we could use bloom filters somehow to determine when no hazard pointer\r\n // can be present in a block, so we can reclaim the entire block in O(k) time...???\r\n // (if we're willing to accept k full, unreclaimable blocks per thread, then we can avoid\r\n // working with individual elements altogether. we can simply check if each HP is in the bloom\r\n // filter for each of c*n blocks (for some constant c), and have some probability of being\r\n // able to reclaim (c-1)*n blocks. then, this procedure will be worst case O(n) time.)\r\n\r\n index[tid*PREFETCH_SIZE_WORDS] = (index[tid*PREFETCH_SIZE_WORDS]+1) % NUMBER_OF_EPOCH_BAGS_CR;\r\n blockbag * const freeable = epochbags[NUMBER_OF_EPOCH_BAGS_CR*tid+index[tid*PREFETCH_SIZE_WORDS]];\r\n if (freeable->getSizeInBlocks() >= scanThreshold) {\r\n\r\n TRACE COUTATOMICTID(\"retiring... we have \"<computeSize()<<\" things waiting to be retired in this epoch bag...\");\r\n // hash all announcements\r\n comparing[tid]->clear();\r\n assert(comparing[tid]->size() == 0);\r\n for (int otherTid=0; otherTid < this->NUM_PROCESSES; ++otherTid) {\r\n int sz = announce[otherTid]->size();\r\n for (int ixHP = 0; ixHP < sz; ++ixHP) {\r\n T* hp = (T*) announce[otherTid]->get(ixHP);\r\n if (hp) {\r\n int oldSize; DEBUG2 oldSize = comparing[tid]->size();\r\n comparing[tid]->insert((T*) hp);\r\n DEBUG2 assert(comparing[tid]->size() <= oldSize + 1); // might not increase size if comparing[tid] already contains this item...\r\n }\r\n }\r\n }\r\n \r\n // check if any nodes (from two epochs ago) are announced (qprotected)\r\n // and swap them to the front of the blockbag.\r\n // once all announced nodes are at the front of the blockbag,\r\n // we can free whole blocks in the remainder of the blockbag.\r\n blockbag_iterator it = freeable->begin();\r\n blockbag_iterator nextswap = freeable->begin();\r\n while (it != freeable->end()) {\r\n if (comparing[tid]->contains(*it)) {\r\n // a hazard pointers points to the item\r\n it.swap(nextswap.getCurr(), nextswap.getIndex());\r\n nextswap++;\r\n }\r\n it++;\r\n }\r\n block * const curr = nextswap.getCurr();\r\n if (curr) {\r\n this->pool->addMoveFullBlocks(tid, freeable, curr);\r\n }\r\n }\r\n \r\n currentBag[tid*PREFETCH_SIZE_WORDS] = freeable;\r\n assert(isQuiescent(tid));\r\n }\r\n \r\n // invoke this at the beginning of each operation that accesses\r\n // objects reclaimed by this epoch manager.\r\n // returns true if the call rotated the epoch bags for thread tid\r\n // (and reclaimed any objects retired two epochs ago).\r\n // otherwise, the call returns false.\r\n // IMPLIES A FULL MEMORY BARRIER\r\n inline bool leaveQuiescentState(const int tid, void * const * const reclaimers, const int numReclaimers) {\r\n SOFTWARE_BARRIER; // prevent any bookkeeping from being moved after this point by the compiler.\r\n bool result = false;\r\n long readEpoch = epoch; // multiple of EPOCH_INCREMENT\r\n assert(!QUIESCENT(readEpoch));\r\n // if our announced epoch is different from the current epoch\r\n const long ann = announcedEpoch[tid*PREFETCH_SIZE_WORDS].load(memory_order_relaxed);\r\n DEBUG2 if (!QUIESCENT(ann)) {\r\n COUTATOMICTID(\"NOT QUIESCENT\"< * const) reclaimers[i])->rotateEpochBags(tid);\r\n }\r\n result = true;\r\n }\r\n // note: readEpoch, when written to announcedEpoch[tid],\r\n // will set the state to non-quiescent and non-neutralized\r\n \r\n // incrementally scan the announced epochs of all threads\r\n int otherTid = checked[tid*PREFETCH_SIZE_WORDS];\r\n if (otherTid >= this->NUM_PROCESSES) {\r\n const int c = ++checked[tid*PREFETCH_SIZE_WORDS];\r\n if (c > MINIMUM_OPERATIONS_BEFORE_NEW_EPOCH_CR) {\r\n __sync_bool_compare_and_swap(&epoch, readEpoch, readEpoch+EPOCH_INCREMENT);\r\n }\r\n } else {\r\n assert(otherTid >= 0);\r\n long otherAnnounce = announcedEpoch[otherTid*PREFETCH_SIZE_WORDS].load(memory_order_relaxed);\r\n if (BITS_EPOCH(otherAnnounce) == readEpoch\r\n || QUIESCENT(otherAnnounce)\r\n || neutralizeOther(tid, otherTid, readEpoch, otherAnnounce)) {\r\n const int c = ++checked[tid*PREFETCH_SIZE_WORDS];\r\n if (c >= this->NUM_PROCESSES && c > MINIMUM_OPERATIONS_BEFORE_NEW_EPOCH_CR) {\r\n __sync_bool_compare_and_swap(&epoch, readEpoch, readEpoch+EPOCH_INCREMENT);\r\n }\r\n }\r\n }\r\n // it is important that we set the announcedEpoch last, because we must\r\n // not be neutralized during some of the preceding steps, or we may\r\n // corrupt the data structure.\r\n // (on x86/64, writes are not moved earlier in program order, so we don't need any membar before this write.)\r\n // (on another arch, we'd have to prevent this write from being moved before the write to checked[].)\r\n assert(isQuiescent(tid));\r\n SOFTWARE_BARRIER;\r\n announcedEpoch[tid*PREFETCH_SIZE_WORDS].store(readEpoch, memory_order_relaxed);\r\n return result;\r\n }\r\n // IN A SCHEME THAT SUPPORTS CRASH RECOVERY, THIS IMPLIES A FULL MEMORY BARRIER IFF THIS MOVES THE THREAD FROM AN ACTIVE STATE TO A QUIESCENT STATE\r\n inline void enterQuiescentState(const int tid) {\r\n const long ann = announcedEpoch[tid*PREFETCH_SIZE_WORDS].load(memory_order_relaxed);\r\n announcedEpoch[tid*PREFETCH_SIZE_WORDS].store(GET_WITH_QUIESCENT(ann), memory_order_relaxed);\r\n assert(isQuiescent(tid));\r\n }\r\n \r\n // for all schemes except reference counting\r\n inline void retire(const int tid, T* p) {\r\n assert(isQuiescent(tid));\r\n currentBag[tid*PREFETCH_SIZE_WORDS]->add(p);\r\n DEBUG2 this->debug->addRetired(tid, 1);\r\n }\r\n\r\n void debugPrintStatus(const int tid) {\r\n// assert(tid >= 0);\r\n// assert(tid < this->NUM_PROCESSES);\r\n// long announce = BITS_EPOCH(announcedEpoch[tid*PREFETCH_SIZE_WORDS].load(memory_order_relaxed))/EPOCH_INCREMENT;\r\n// std::cout<<\"announce=\"<computeSize();\r\n// }\r\n }\r\n\r\n reclaimer_debraplus(const int numProcesses, Pool *_pool, debugInfo * const _debug, RecoveryMgr * const _recoveryMgr = NULL)\r\n : reclaimer_interface(numProcesses, _pool, _debug, _recoveryMgr) {\r\n VERBOSE DEBUG COUTATOMIC(\"constructor reclaimer_debraplus helping=\"<shouldHelp()<recoveryMgr->neutralizeSignal<recoveryMgr->neutralizeSignal)) {\r\n COUTATOMIC(\"error adding signal to signal set\"<*[NUMBER_OF_EPOCH_BAGS_CR*numProcesses];\r\n currentBag = new blockbag*[numProcesses*PREFETCH_SIZE_WORDS];\r\n index = new long[numProcesses*PREFETCH_SIZE_WORDS];\r\n announcedEpoch = new atomic_long[numProcesses*PREFETCH_SIZE_WORDS];\r\n checked = new long[numProcesses*PREFETCH_SIZE_WORDS];\r\n announce = new AtomicArrayList*[numProcesses];\r\n comparing = new hashset_new*[numProcesses];\r\n for (int tid=0;tid(tid, this->pool->blockpools[tid]);\r\n }\r\n currentBag[tid*PREFETCH_SIZE_WORDS] = epochbags[NUMBER_OF_EPOCH_BAGS_CR*tid];\r\n index[tid*PREFETCH_SIZE_WORDS] = 0;\r\n announcedEpoch[tid*PREFETCH_SIZE_WORDS].store(GET_WITH_QUIESCENT(0), memory_order_relaxed);\r\n checked[tid*PREFETCH_SIZE_WORDS] = 0;\r\n announce[tid] = new AtomicArrayList(MAX_PROTECT_EVEN_IF_QUIESCENT);\r\n comparing[tid] = new hashset_new(numProcesses*MAX_PROTECT_EVEN_IF_QUIESCENT);\r\n }\r\n }\r\n ~reclaimer_debraplus() {\r\n VERBOSE DEBUG COUTATOMIC(\"destructor reclaimer_debraplus\"<NUM_PROCESSES;++tid) {\r\n // move contents of all bags into pool\r\n for (int i=0;ipool->addMoveAll(tid, epochbags[NUMBER_OF_EPOCH_BAGS_CR*tid+i]);\r\n delete epochbags[NUMBER_OF_EPOCH_BAGS_CR*tid+i];\r\n }\r\n delete comparing[tid];\r\n delete announce[tid];\r\n }\r\n delete[] announce;\r\n delete[] epochbags;\r\n delete[] index;\r\n delete[] currentBag;\r\n delete[] announcedEpoch;\r\n delete[] checked;\r\n delete[] comparing;\r\n }\r\n\r\n};\r\n\r\n#endif\r\n\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/recordmgr/reclaimer_hazardptr.h", "content": "/**\r\n * C++ record manager implementation (PODC 2015) by Trevor Brown.\r\n * \r\n * Copyright (C) 2015 Trevor Brown\r\n *\r\n */\r\n\r\n#ifndef RECLAIM_HAZARDPTR_STACK_H\r\n#define\tRECLAIM_HAZARDPTR_STACK_H\r\n\r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \"blockbag.h\"\r\n#include \"plaf.h\"\r\n#include \"allocator_interface.h\"\r\n#include \"hashtable.h\"\r\n#include \"reclaimer_interface.h\"\r\n#include \"arraylist.h\"\r\nusing namespace std;\r\nusing namespace hashset_namespace;\r\n\r\n#define MAX_HAZARDPTRS_PER_THREAD 16\r\n\r\ntemplate >\r\nclass reclaimer_hazardptr : public reclaimer_interface {\r\nprivate:\r\n AtomicArrayList **announce; // announce[tid] = set of announced hazard pointers for thread tid\r\n ArrayList **retired; // retired[tid] = set of retired objects for thread tid\r\n hashset_new **comparing; // comparing[tid] = set of announced hazard pointers for ALL threads, as collected by thread tid during it's last retire(tid, ...) call\r\n \r\n // number of elements that retired[tid] must contain\r\n // before we scan hazard pointers to determine\r\n // which elements of retired[tid] can be deallocated.\r\n // to get amortized constant scanning time per object,\r\n // this must be nk+Omega(nk), where\r\n // n = number of threads and\r\n // k = max number of hazard pointers a thread can hold at once\r\n const int scanThreshold;\r\n \r\npublic:\r\n template\r\n struct rebind {\r\n typedef reclaimer_hazardptr<_Tp1, Pool> other;\r\n };\r\n template\r\n struct rebind2 {\r\n typedef reclaimer_hazardptr<_Tp1, _Tp2> other;\r\n };\r\n \r\n inline static bool shouldHelp() {\r\n return false;\r\n }\r\n \r\n bool isProtected(const int tid, T * const obj) {\r\n return announce[tid]->contains(obj);\r\n }\r\n bool static isQProtected(const int tid, T * const obj) {\r\n return false;\r\n }\r\n inline static bool isQuiescent(const int tid) {\r\n return true;\r\n } \r\n \r\n // for hazard pointers (and counting references from threads)\r\n inline bool protect(const int tid, T * const obj, CallbackType notRetiredCallback, CallbackArg callbackArg, bool memoryBarrier = true) {\r\n TRACE std::cout<<\"reclaimer_hazardptr::protect(tid=\"<size();\r\n// DEBUG if (sizeof(T) < 80 /* is a node */) assert(!announce[tid]->contains(obj));\r\n announce[tid]->add(obj);\r\n if (memoryBarrier) __sync_synchronize(); // prevent retired from being read before we set a hazard pointer to obj\r\n DEBUG2 assert(isProtected(tid, obj)); //announce[tid]->contains(obj));\r\n DEBUG2 assert(size + 1 == announce[tid]->size());\r\n// SOFTWARE_BARRIER;\r\n if (notRetiredCallback(callbackArg)) {\r\n// SOFTWARE_BARRIER;\r\n TRACE std::cout<<\"notRetiredCallback returns true\"<size() <= MAX_HAZARDPTRS_PER_THREAD);\r\n DEBUG2 assert(isProtected(tid, obj));\r\n// SOFTWARE_BARRIER;\r\n assert(isProtected(tid, obj));\r\n return true;\r\n } else {\r\n TRACE std::cout<<\"notRetiredCallback returns false\"<size();\r\n announce[tid]->erase(obj);\r\n// DEBUG if (sizeof(T) < 80 /* is a node */) assert(!announce[tid]->contains(obj));\r\n DEBUG2 assert(size - 1 == announce[tid]->size());\r\n// SOFTWARE_BARRIER;\r\n }\r\n inline bool qProtect(const int tid, T * const obj, CallbackType notRetiredCallback, CallbackArg callbackArg, bool memoryBarrier = true) {\r\n TRACE std::cout<<\"reclaimer_debraplus::qProtect(tid=\"<clear();\r\n// __sync_synchronize();\r\n// announce[tid]->clearWithoutFreeingElements();\r\n DEBUG2 assert(announce[tid]->size() == 0);\r\n DEBUG2 assert(announce[tid]->isEmpty());\r\n// SOFTWARE_BARRIER;\r\n }\r\n inline static bool leaveQuiescentState(const int tid, void * const * const reclaimers, const int numReclaimers) {\r\n TRACE std::cout<<\"reclaimer_hazardptr::leaveQuiescentState(tid=\"< 0) {\r\n int c = x % base;\r\n if (c < 10) os<<(char)(c+(int)'0');\r\n else if (c < 10+26) os<<(char)(c-10+(int)'a');\r\n else os<<(char)(c-10-26+(int)'A');\r\n x /= base;\r\n }\r\n return os.str();\r\n }\r\n \r\n inline void retire(const int tid, T* p) {\r\n TRACE std::cout<<\"reclaimer_hazardptr::retire(tid=\"<debug->addRetired(tid, 1);\r\n retired[tid]->add(p);\r\n \r\n // if the retired bag is sufficiently large\r\n if (retired[tid]->isFull()) {\r\n// __sync_synchronize(); // not necessary, since there is a membar implied by the update cas between here and the marked bit that makes the retired predicate return true... (it follows that the retired predicate for a node u will see marked and return true if it executes when we are performing retire(u).)\r\n \r\n// TRACE std::cout<<\"retiring... we have \"<size()<<\" things waiting to be retired (#hps=\"<size()<<\")...\";\r\n// // hash all announcements\r\n// int totalSize = 0;\r\n// int sizes[MAX_TID_POW2];\r\n// for (int otherTid=0; otherTid < this->NUM_PROCESSES; ++otherTid) {\r\n// sizes[otherTid] = announce[tid]->size();\r\n// totalSize += sizes[otherTid];\r\n// }\r\n// hashset_new hset = hashset_new(totalSize);\r\n// for (int otherTid=0; otherTid < this->NUM_PROCESSES; ++otherTid) {\r\n// for (int i=0;iget(i));\r\n// }\r\n// }\r\n// \r\n// // iterate over all items in retired[tid]\r\n// TRACE std::cout<<\"retiring... we have \"<size()<<\" things waiting to be retired (#hps=\"<size()<<\", totalSize=\"<size();++ix) {\r\n// TRACE std::cout<<\" \"<get(ix))<<\"=\"<<(hset.contains(retired[tid]->get(ix))?\"1\":\"0\");\r\n// if (!hset.contains(retired[tid]->get(ix))) {\r\n// // no hazard pointers point to the item, so we send it to the pool\r\n// this->pool->add(tid, retired[tid]->get(ix));\r\n// // now we remove the item from retired[tid] and\r\n// // adjust ix to continue where we left off\r\n// retired[tid]->erase(ix);\r\n// --ix;\r\n// }\r\n// }\r\n// TRACE std::cout<<\" afterwards, we have \"<size()<<\" things waiting to be retired...\"<size()<<\" things waiting to be retired (THIS thread #hps=\"<size()<<\")...\";\r\n// for (int ix=0;ixsize();) {\r\n// // check if retired[tid]->data[ix] is in any set of hazard pointers\r\n// bool found = false;\r\n// for (int otherTid=0;otherTidNUM_PROCESSES;++otherTid) {\r\n// int sz = announce[otherTid]->size();\r\n// for (int ixHP=0;ixHPget(ix) == announce[otherTid]->get(ixHP)) {\r\n// found = true;\r\n// // break out of both loops\r\n// otherTid = this->NUM_PROCESSES;\r\n// break;\r\n// }\r\n// }\r\n// }\r\n// if (!found) {\r\n// // no hazard pointers point to the item, so we send it to the pool\r\n// this->pool->add(tid, retired[tid]->get(ix));\r\n// // now we remove the item from retired[tid]\r\n// retired[tid]->erase(ix);\r\n// } else {\r\n// ++ix; // we didn't erase, so we need to move on to the next element\r\n// }\r\n// }\r\n// TRACE std::cout<<\" afterwards, we have \"<size()<<\" things waiting to be retired...\"<size()<<\" things waiting to be retired (THIS thread #hps=\"<size()<<\")...\";\r\n // hash all announcements\r\n comparing[tid]->clear();\r\n assert(comparing[tid]->size() == 0);\r\n for (int otherTid=0; otherTid < this->NUM_PROCESSES; ++otherTid) {\r\n int sz = announce[otherTid]->size();\r\n assert(sz < MAX_HAZARDPTRS_PER_THREAD);\r\n for (int ixHP=0;ixHPsize();\r\n comparing[tid]->insert(announce[otherTid]->get(ixHP));\r\n DEBUG2 assert(comparing[tid]->size() <= oldSize + 1); // might not increase size if comparing[tid] already contains this item...\r\n }\r\n }\r\n for (int ix=0;ixsize();) {\r\n // check if retired[tid]->data[ix] is in any set of hazard pointers\r\n if (!comparing[tid]->contains(retired[tid]->get(ix))) {\r\n // no hazard pointers point to the item, so we send it to the pool\r\n this->pool->add(tid, retired[tid]->get(ix));\r\n // now we remove the item from retired[tid]\r\n retired[tid]->erase(ix);\r\n } else {\r\n ++ix; // we didn't erase, so we need to move on to the next element\r\n }\r\n }\r\n TRACE std::cout<<\" afterwards, we have \"<size()<<\" things waiting to be retired...\"<isFull());\r\n }\r\n }\r\n\r\n void debugPrintStatus(const int tid) {\r\n// assert(tid >= 0);\r\n// assert(tid < this->NUM_PROCESSES);\r\n }\r\n\r\n reclaimer_hazardptr(const int numProcesses, Pool *_pool, debugInfo * const _debug, RecoveryMgr * const _recoveryMgr = NULL)\r\n : scanThreshold(5*numProcesses*MAX_HAZARDPTRS_PER_THREAD),\r\n reclaimer_interface(numProcesses, _pool, _debug, _recoveryMgr) {\r\n VERBOSE DEBUG std::cout<<\"constructor reclaimer_hazardptr\"<*[numProcesses];\r\n retired = new ArrayList*[numProcesses];\r\n comparing = new hashset_new*[numProcesses];\r\n for (int tid=0;tid(MAX_HAZARDPTRS_PER_THREAD);\r\n retired[tid] = new ArrayList(scanThreshold);\r\n comparing[tid] = new hashset_new(numProcesses*MAX_HAZARDPTRS_PER_THREAD);\r\n }\r\n }\r\n ~reclaimer_hazardptr() {\r\n VERBOSE DEBUG std::cout<<\"destructor reclaimer_hazardptr\"<NUM_PROCESSES;++tid) {\r\n int sz = retired[tid]->size();\r\n for (int ix=0;ixpool->add(tid, retired[tid]->get(ix));\r\n }\r\n delete announce[tid];\r\n delete retired[tid];\r\n delete comparing[tid];\r\n }\r\n delete[] announce;\r\n delete[] retired;\r\n delete[] comparing;\r\n }\r\n\r\n}; // end class\r\n\r\n#endif\r\n\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/recordmgr/reclaimer_interface.h", "content": "/**\r\n * C++ record manager implementation (PODC 2015) by Trevor Brown.\r\n * \r\n * Copyright (C) 2015 Trevor Brown\r\n *\r\n */\r\n\r\n#ifndef RECLAIM_INTERFACE_H\r\n#define\tRECLAIM_INTERFACE_H\r\n\r\n#include \"recovery_manager.h\"\r\n#include \"pool_interface.h\"\r\n#include \"globals.h\"\r\n#include \r\n#include \r\nusing namespace std;\r\n\r\ntemplate \r\nstruct set_of_bags {\r\n blockbag * const * const bags;\r\n const int numBags;\r\n};\r\n\r\ntemplate >\r\nclass reclaimer_interface {\r\npublic:\r\n#ifndef __CYGWIN__\r\n RecoveryMgr * recoveryMgr;\r\n#endif\r\n debugInfo * const debug;\r\n \r\n const int NUM_PROCESSES;\r\n Pool *pool;\r\n\r\n template\r\n struct rebind {\r\n typedef reclaimer_interface<_Tp1, Pool> other;\r\n };\r\n template\r\n struct rebind2 {\r\n typedef reclaimer_interface<_Tp1, _Tp2> other;\r\n };\r\n \r\n long long getSizeInNodes() { return 0; }\r\n string getSizeString() { return \"\"; }\r\n\r\n inline static bool quiescenceIsPerRecordType() { return true; }\r\n inline static bool shouldHelp() { return true; } // FOR DEBUGGING PURPOSES\r\n inline static bool supportsCrashRecovery() { return false; }\r\n inline bool isProtected(const int tid, T * const obj);\r\n inline bool isQProtected(const int tid, T * const obj);\r\n inline static bool isQuiescent(const int tid) {\r\n COUTATOMICTID(\"reclaimer_interface::isQuiescent(tid) is not implemented!\"< must be idempotent,\r\n * and must unprotect all objects protected by calls to protectObject.\r\n * it must NOT unprotect any object protected by a call to\r\n * protectObjectEvenAfterRestart.\r\n */\r\n inline void enterQuiescentState(const int tid);\r\n inline bool leaveQuiescentState(const int tid, void * const * const reclaimers, const int numReclaimers);\r\n inline void rotateEpochBags(const int tid);\r\n\r\n // for all schemes except reference counting\r\n inline void retire(const int tid, T* p);\r\n inline void unretireLast(const int tid) {}\r\n \r\n inline void initThread(const int tid) {}\r\n inline void deinitThread(const int tid) {}\r\n void debugPrintStatus(const int tid);\r\n \r\n reclaimer_interface(const int numProcesses, Pool *_pool, debugInfo * const _debug, RecoveryMgr * const _recoveryMgr = NULL)\r\n#ifndef __CYGWIN__\r\n : recoveryMgr(_recoveryMgr)\r\n#endif\r\n , debug(_debug)\r\n , NUM_PROCESSES(numProcesses)\r\n , pool(_pool) {\r\n VERBOSE DEBUG COUTATOMIC(\"constructor reclaimer_interface\"<\r\n#include \r\n#include \"pool_interface.h\"\r\n#include \"reclaimer_interface.h\"\r\nusing namespace std;\r\n\r\ntemplate >\r\nclass reclaimer_none : public reclaimer_interface {\r\nprivate:\r\npublic:\r\n template\r\n struct rebind {\r\n typedef reclaimer_none<_Tp1, Pool> other;\r\n };\r\n template\r\n struct rebind2 {\r\n typedef reclaimer_none<_Tp1, _Tp2> other;\r\n };\r\n \r\n string getSizeString() { return \"no reclaimer\"; }\r\n inline static bool shouldHelp() {\r\n return true;\r\n }\r\n \r\n inline static bool isQuiescent(const int tid) {\r\n return true;\r\n }\r\n inline static bool isProtected(const int tid, T * const obj) {\r\n return true;\r\n }\r\n inline static bool isQProtected(const int tid, T * const obj) {\r\n return false;\r\n }\r\n \r\n // for hazard pointers (and reference counting)\r\n inline static bool protect(const int tid, T * const obj, CallbackType notRetiredCallback, CallbackArg callbackArg, bool memoryBarrier = true) {\r\n return true;\r\n }\r\n inline static void unprotect(const int tid, T * const obj) {}\r\n inline static bool qProtect(const int tid, T * const obj, CallbackType notRetiredCallback, CallbackArg callbackArg, bool memoryBarrier = true) {\r\n return true;\r\n }\r\n inline static void qUnprotectAll(const int tid) {}\r\n \r\n // rotate the epoch bags and reclaim any objects retired two epochs ago.\r\n inline static void rotateEpochBags(const int tid) {\r\n }\r\n // invoke this at the beginning of each operation that accesses\r\n // objects reclaimed by this epoch manager.\r\n // returns true if the call rotated the epoch bags for thread tid\r\n // (and reclaimed any objects retired two epochs ago).\r\n // otherwise, the call returns false.\r\n inline static bool leaveQuiescentState(const int tid, void * const * const reclaimers, const int numReclaimers) {\r\n return false;\r\n }\r\n inline static void enterQuiescentState(const int tid) {\r\n }\r\n \r\n // for all schemes except reference counting\r\n inline static void retire(const int tid, T* p) {\r\n }\r\n\r\n void debugPrintStatus(const int tid) {\r\n }\r\n\r\n// set_of_bags getBlockbags() {\r\n// set_of_bags empty = {.bags = NULL, .numBags = 0};\r\n// return empty;\r\n// }\r\n// \r\n// void getOldestTwoBlockbags(const int tid, blockbag ** oldest, blockbag ** secondOldest) {\r\n// *oldest = *secondOldest = NULL;\r\n// }\r\n// \r\n// int getOldestBlockbagIndexOffset(const int tid) {\r\n// return -1;\r\n// }\r\n \r\n void getSafeBlockbags(const int tid, blockbag ** bags) {\r\n bags[0] = NULL;\r\n }\r\n \r\n reclaimer_none(const int numProcesses, Pool *_pool, debugInfo * const _debug, RecoveryMgr * const _recoveryMgr = NULL)\r\n : reclaimer_interface(numProcesses, _pool, _debug, _recoveryMgr) {\r\n VERBOSE DEBUG std::cout<<\"constructor reclaimer_none\"<\r\n#include \r\n#include \r\n#include \r\n#include \"blockbag.h\"\r\n#include \"plaf.h\"\r\n#ifdef USE_DEBUGCOUNTERS\r\n #include \"debugcounter.h\"\r\n#endif\r\n#include \"allocator_interface.h\"\r\n#include \"reclaimer_interface.h\"\r\n#ifdef BST\r\n #include \"node.h\"\r\n #include \"scxrecord.h\"\r\n#elif defined KCAS_MAXK\r\n #include \"kcas.h\"\r\n#else\r\n #error ONLY SUPPORTS BST(main.cpp) and KCAS(ubench.cpp)\r\n#endif\r\nusing namespace std;\r\n\r\n#include \r\n\r\ntemplate M get_member_type(M T::*);\r\ntemplate T get_class_type(M T::*);\r\n\r\ntemplate \r\nconstexpr std::size_t offset_of()\r\n{\r\n return reinterpret_cast(&(((T*)0)->*M));\r\n}\r\n\r\n#define OFFSET_OF(m) offset_of()\r\n\r\n#define comma ,\r\n\r\n__thread long long rcuthrFreesNode = 0; // for RCU THREADS ONLY\r\n__thread long long rcuthrFreesDescriptor = 0; // for RCU THREADS ONLY\r\nlong long freesNode = 0;\r\nlong long freesDescriptor = 0;\r\n\r\n#if defined BST || defined BST_THROWAWAY\r\n void rcuCallback_Node(struct rcu_head *rcu) {\r\n Node * n = (Node *)\r\n (((char*) rcu) - OFFSET_OF(\r\n &Node::rcuHeadField));\r\n if (++rcuthrFreesNode == 1<<10) {\r\n __sync_fetch_and_add(&freesNode, rcuthrFreesNode);\r\n rcuthrFreesNode = 0;\r\n }\r\n free(n);\r\n }\r\n #ifdef BST_THROWAWAY\r\n void rcuCallback_SCXRecord(struct rcu_head *rcu) {\r\n SCXRecord * n = (SCXRecord *)\r\n (((char*) rcu) - OFFSET_OF(\r\n &SCXRecord::rcuHeadField));\r\n if (++rcuthrFreesDescriptor == 1<<10) {\r\n __sync_fetch_and_add(&freesDescriptor, rcuthrFreesDescriptor);\r\n rcuthrFreesDescriptor = 0;\r\n }\r\n free(n);\r\n }\r\n #endif\r\n#elif defined KCAS_MAXK\r\n void rcuCallback_kcasdesc(struct rcu_head *rcu) {\r\n kcasdesc_t * n = (kcasdesc_t *)\r\n (((char*) rcu) - OFFSET_OF(\r\n &kcasdesc_t::rcuHeadField));\r\n if (++rcuthrFreesDescriptor == 1<<10) {\r\n __sync_fetch_and_add(&freesDescriptor, rcuthrFreesDescriptor);\r\n rcuthrFreesDescriptor = 0;\r\n }\r\n free(n);\r\n }\r\n void rcuCallback_rdcssdesc(struct rcu_head *rcu) {\r\n rdcssdesc_t * n = (rdcssdesc_t *)\r\n (((char*) rcu) - OFFSET_OF(\r\n &rdcssdesc_t::rcuHeadField));\r\n if (++rcuthrFreesNode == 1<<10) {\r\n __sync_fetch_and_add(&freesNode, rcuthrFreesNode);\r\n rcuthrFreesNode = 0;\r\n }\r\n free(n);\r\n }\r\n#endif\r\n\r\n//template \r\n//void rcuCallback(struct rcu_head *rcu) {\r\n// T * n = (T *) (((char *) rcu) - OFFSET_OF(&T::rcuHeadField));\r\n// free(n);\r\n//}\r\n\r\n__thread bool calledRCULock = false;\r\n__thread bool rcuInitialized = false;\r\n\r\ntemplate >\r\nclass reclaimer_rcu : public reclaimer_interface {\r\nprotected:\r\n \r\npublic:\r\n template\r\n struct rebind {\r\n typedef reclaimer_rcu<_Tp1, Pool> other;\r\n };\r\n template\r\n struct rebind2 {\r\n typedef reclaimer_rcu<_Tp1, _Tp2> other;\r\n };\r\n \r\n long long getSizeInNodes() {\r\n long long sum = 0;\r\n return sum;\r\n }\r\n string getSizeString() {\r\n stringstream ss;\r\n ss<rcuHeadField, rcuCallback);\r\n#if defined BST || defined BST_THROWAWAY\r\n if (sizeof(*p) == sizeof(Node)) {\r\n call_rcu(&p->rcuHeadField, rcuCallback_Node);\r\n#ifdef BST_THROWAWAY\r\n } else if (sizeof(*p) == sizeof(SCXRecord)) {\r\n call_rcu(&p->rcuHeadField, rcuCallback_SCXRecord);\r\n#endif\r\n }\r\n#elif defined KCAS_MAXK\r\n if (sizeof(*p) == sizeof(kcasdesc_t)) {\r\n call_rcu(&p->rcuHeadField, rcuCallback_kcasdesc);\r\n } else if (sizeof(*p) == sizeof(rdcssdesc_t)) {\r\n call_rcu(&p->rcuHeadField, rcuCallback_rdcssdesc);\r\n }\r\n#endif\r\n }\r\n\r\n void debugPrintStatus(const int tid) {\r\n if (freesNode) std::cout<<\"freesNode=\"< * const _recoveryMgr = NULL)\r\n : reclaimer_interface(numProcesses, _pool, _debug, _recoveryMgr) {\r\n rcu_init();\r\n }\r\n ~reclaimer_rcu() {}\r\n};\r\n\r\n#endif\r\n\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/recordmgr/record_manager.h", "content": "/**\r\n * C++ record manager implementation (PODC 2015) by Trevor Brown.\r\n * \r\n * Copyright (C) 2015 Trevor Brown\r\n *\r\n */\r\n\r\n#ifndef RECORD_MANAGER_H\r\n#define\tRECORD_MANAGER_H\r\n\r\n#include \r\n#include \"globals.h\"\r\n#include \"record_manager_single_type.h\"\r\n\r\n#include \r\n#include \r\n#include \r\n#include \r\nusing namespace std;\r\n\r\ninline CallbackReturn callbackReturnTrue(CallbackArg arg) {\r\n return true;\r\n}\r\n\r\n// compile time check for duplicate template parameters\r\n// compare first with rest to find any duplicates\r\ntemplate void check_duplicates(void) {}\r\ntemplate \r\nvoid check_duplicates(void) {\r\n if (typeid(T) == typeid(First)) {\r\n throw logic_error(\"duplicate template arguments provided to RecordManagerSet\");\r\n }\r\n check_duplicates();\r\n}\r\n\r\n// base case: empty template\r\n// this is a compile time check for invalid arguments\r\ntemplate \r\nclass RecordManagerSet {\r\npublic:\r\n RecordManagerSet(const int numProcesses, RecoveryMgr * const _recoveryMgr) {}\r\n template \r\n record_manager_single_type * get(T * const recordType) {\r\n throw logic_error(\"invalid type passed to RecordManagerSet::get()\");\r\n return NULL;\r\n }\r\n void clearCounters(void) {}\r\n void registerThread(const int tid) {}\r\n void unregisterThread(const int tid) {}\r\n void printStatus() {}\r\n inline void qUnprotectAll(const int tid) {}\r\n inline void getReclaimers(const int tid, void ** const reclaimers, int index) {}\r\n inline void enterQuiescentState(const int tid) {}\r\n inline void leaveQuiescentStateForEach(const int tid) {}\r\n inline void leaveQuiescentState(const int tid, const bool callForEach) {}\r\n};\r\n\r\n// \"recursive\" case\r\ntemplate \r\nclass RecordManagerSet : RecordManagerSet {\r\n record_manager_single_type * const mgr;\r\npublic:\r\n RecordManagerSet(const int numProcesses, RecoveryMgr * const _recoveryMgr)\r\n : RecordManagerSet(numProcesses, _recoveryMgr)\r\n , mgr(new record_manager_single_type(numProcesses, _recoveryMgr))\r\n {\r\n //cout<<\"RecordManagerSet with First=\"<(); // check if first is in {rest...}\r\n }\r\n ~RecordManagerSet() {\r\n delete mgr;\r\n // note: should automatically call the parent class' destructor afterwards\r\n }\r\n // note: the compiled code for get() should be a single read and return statement\r\n template\r\n inline record_manager_single_type * get(T * const recordType) {\r\n if (typeid(First) == typeid(T)) {\r\n //cout<<\"MATCH: typeid(First)=\"< *) mgr;\r\n } else {\r\n //cout<<\"NO MATCH: typeid(First)=\"< *) this)->get(recordType);\r\n }\r\n }\r\n // note: recursion should be compiled out\r\n void clearCounters(void) {\r\n mgr->clearCounters();\r\n ((RecordManagerSet *) this)->clearCounters();\r\n }\r\n void registerThread(const int tid) {\r\n mgr->initThread(tid);\r\n ((RecordManagerSet *) this)->registerThread(tid);\r\n }\r\n void unregisterThread(const int tid) {\r\n mgr->deinitThread(tid);\r\n ((RecordManagerSet *) this)->unregisterThread(tid);\r\n }\r\n void printStatus() {\r\n mgr->printStatus();\r\n ((RecordManagerSet *) this)->printStatus();\r\n }\r\n inline void qUnprotectAll(const int tid) {\r\n mgr->qUnprotectAll(tid);\r\n ((RecordManagerSet *) this)->qUnprotectAll(tid);\r\n }\r\n inline void getReclaimers(const int tid, void ** const reclaimers, int index) {\r\n reclaimers[index] = mgr->reclaim;\r\n ((RecordManagerSet *) this)->getReclaimers(tid, reclaimers, 1+index);\r\n }\r\n inline void enterQuiescentState(const int tid) {\r\n mgr->enterQuiescentState(tid);\r\n ((RecordManagerSet *) this)->enterQuiescentState(tid);\r\n }\r\n inline void leaveQuiescentStateForEach(const int tid) {\r\n mgr->leaveQuiescentState(tid, NULL, 0);\r\n ((RecordManagerSet *) this)->leaveQuiescentStateForEach(tid);\r\n }\r\n inline void leaveQuiescentState(const int tid, const bool callForEach) {\r\n if (callForEach) {\r\n leaveQuiescentStateForEach(tid);\r\n } else {\r\n void * reclaimers[1+sizeof...(Rest)];\r\n getReclaimers(tid, reclaimers, 0);\r\n get((First *) NULL)->leaveQuiescentState(tid, reclaimers, 1+sizeof...(Rest));\r\n __sync_synchronize(); // memory barrier needed (only) for epoch based schemes at the moment...\r\n }\r\n }\r\n};\r\n\r\ntemplate \r\nclass record_manager {\r\nprotected:\r\n typedef record_manager SelfType;\r\n RecordManagerSet * rmset;\r\n \r\n int init[MAX_TID_POW2] = {0,};\r\n\r\npublic:\r\n const int NUM_PROCESSES;\r\n RecoveryMgr * const recoveryMgr;\r\n \r\n record_manager(const int numProcesses, const int _neutralizeSignal)\r\n : NUM_PROCESSES(numProcesses)\r\n , recoveryMgr(new RecoveryMgr(numProcesses, _neutralizeSignal, this))\r\n {\r\n rmset = new RecordManagerSet(numProcesses, (RecoveryMgr *) recoveryMgr);\r\n }\r\n ~record_manager() {\r\n delete recoveryMgr;\r\n delete rmset;\r\n }\r\n void initThread(const int tid) {\r\n if (init[tid]) return; else init[tid] = !init[tid];\r\n\r\n rmset->registerThread(tid);\r\n recoveryMgr->initThread(tid);\r\n enterQuiescentState(tid);\r\n }\r\n void deinitThread(const int tid) {\r\n if (!init[tid]) return; else init[tid] = !init[tid];\r\n\r\n rmset->unregisterThread(tid);\r\n }\r\n void clearCounters() {\r\n rmset->clearCounters();\r\n }\r\n void printStatus(void) {\r\n rmset->printStatus();\r\n }\r\n template \r\n debugInfo * getDebugInfo(T * const recordType) {\r\n return &rmset->get((T *) NULL)->debugInfoRecord;\r\n }\r\n template\r\n inline record_manager_single_type * get(T * const recordType) {\r\n return rmset->get((T *) NULL);\r\n }\r\n \r\n // for hazard pointers\r\n\r\n template \r\n inline bool isProtected(const int tid, T * const obj) {\r\n return rmset->get((T *) NULL)->isProtected(tid, obj);\r\n }\r\n \r\n template \r\n inline bool protect(const int tid, T * const obj, CallbackType notRetiredCallback, CallbackArg callbackArg, bool hintMemoryBarrier = true) {\r\n return rmset->get((T *) NULL)->protect(tid, obj, notRetiredCallback, callbackArg, hintMemoryBarrier);\r\n }\r\n \r\n template \r\n inline void unprotect(const int tid, T * const obj) {\r\n rmset->get((T *) NULL)->unprotect(tid, obj);\r\n }\r\n \r\n // for DEBRA+\r\n \r\n // warning: qProtect must be reentrant and lock-free (i.e., async-signal-safe)\r\n template \r\n inline bool qProtect(const int tid, T * const obj, CallbackType notRetiredCallback, CallbackArg callbackArg, bool hintMemoryBarrier = true) {\r\n return rmset->get((T *) NULL)->qProtect(tid, obj, notRetiredCallback, callbackArg, hintMemoryBarrier);\r\n }\r\n \r\n template \r\n inline bool isQProtected(const int tid, T * const obj) {\r\n return rmset->get((T *) NULL)->isQProtected(tid, obj);\r\n }\r\n \r\n inline void qUnprotectAll(const int tid) {\r\n assert(!Reclaim::supportsCrashRecovery() || isQuiescent(tid));\r\n rmset->qUnprotectAll(tid);\r\n }\r\n\r\n // for epoch based reclamation\r\n inline bool isQuiescent(const int tid) {\r\n return rmset->get((RecordTypesFirst *) NULL)->isQuiescent(tid); // warning: if quiescence information is logically shared between all types, with the actual data being associated only with the first type (as it is here), then isQuiescent will return inconsistent results if called in functions that recurse on the template argument list in this class.\r\n }\r\n inline void enterQuiescentState(const int tid) {\r\n// VERBOSE DEBUG2 COUTATOMIC(\"record_manager_single_type::enterQuiescentState(tid=\"<enterQuiescentState(tid);\r\n } else {\r\n // only call enterQuiescentState for one object type\r\n// std::cout<<\"setting quiescent state for just one record type: \"<get((RecordTypesFirst *) NULL)->enterQuiescentState(tid);\r\n }\r\n }\r\n inline void leaveQuiescentState(const int tid) {\r\n// assert(isQuiescent(tid));\r\n// VERBOSE DEBUG2 COUTATOMIC(\"record_manager_single_type::leaveQuiescentState(tid=\"<leaveQuiescentState(tid, Reclaim::quiescenceIsPerRecordType());\r\n }\r\n\r\n // for algorithms that retire nodes before a deletion is linearized\r\n template \r\n inline void unretireLast(const int tid) {\r\n assert(!Reclaim::supportsCrashRecovery() || isQuiescent(tid));\r\n rmset->get((T *) NULL)->unretireLast(tid);\r\n }\r\n\r\n // for all schemes\r\n template \r\n inline void retire(const int tid, T * const p) {\r\n assert(!Reclaim::supportsCrashRecovery() || isQuiescent(tid));\r\n rmset->get((T *) NULL)->retire(tid, p);\r\n }\r\n\r\n template \r\n inline T * allocate(const int tid) {\r\n assert(!Reclaim::supportsCrashRecovery() || isQuiescent(tid));\r\n return rmset->get((T *) NULL)->allocate(tid);\r\n }\r\n \r\n // optional function which can be used if it is safe to call free()\r\n template \r\n inline void deallocate(const int tid, T * const p) {\r\n assert(!Reclaim::supportsCrashRecovery() || isQuiescent(tid));\r\n rmset->get((T *) NULL)->deallocate(tid, p);\r\n }\r\n\r\n inline static bool shouldHelp() { // FOR DEBUGGING PURPOSES\r\n return Reclaim::shouldHelp();\r\n }\r\n inline static bool supportsCrashRecovery() {\r\n return Reclaim::supportsCrashRecovery();\r\n }\r\n};\r\n\r\n#endif\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/recordmgr/record_manager_single_type.h", "content": "/**\r\n * C++ record manager implementation (PODC 2015) by Trevor Brown.\r\n * \r\n * Copyright (C) 2015 Trevor Brown\r\n *\r\n */\r\n\r\n#ifndef RECORD_MANAGER_SINGLE_TYPE_H\r\n#define\tRECORD_MANAGER_SINGLE_TYPE_H\r\n\r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \r\n\r\n#include \"plaf.h\"\r\n#include \"debug_info.h\"\r\n#include \"globals.h\"\r\n\r\n#include \"recovery_manager.h\"\r\n\r\n#include \"allocator_interface.h\"\r\n#include \"allocator_bump.h\"\r\n#include \"allocator_new.h\"\r\n#include \"allocator_new_segregated.h\"\r\n#include \"allocator_once.h\"\r\n\r\n#include \"pool_interface.h\"\r\n#include \"pool_none.h\"\r\n#include \"pool_perthread_and_shared.h\"\r\n\r\n#include \"reclaimer_interface.h\"\r\n#include \"reclaimer_none.h\"\r\n#include \"reclaimer_debra.h\"\r\n#include \"reclaimer_debraplus.h\"\r\n#include \"reclaimer_hazardptr.h\"\r\n#ifdef USE_RECLAIMER_RCU\r\n#include \"reclaimer_rcu.h\"\r\n#endif\r\n\r\nusing namespace std;\r\n\r\n// maybe Record should be a size\r\ntemplate \r\nclass record_manager_single_type {\r\nprotected:\r\n typedef Record* record_pointer;\r\n\r\n typedef typename Alloc::template rebind::other classAlloc;\r\n typedef typename Pool::template rebind2::other classPool;\r\n typedef typename Reclaim::template rebind2::other classReclaim;\r\n \r\npublic:\r\n classAlloc *alloc;\r\n classPool *pool;\r\n classReclaim *reclaim;\r\n \r\n const int NUM_PROCESSES;\r\n debugInfo debugInfoRecord;\r\n RecoveryMgr * const recoveryMgr;\r\n\r\n record_manager_single_type(const int numProcesses, RecoveryMgr * const _recoveryMgr)\r\n : NUM_PROCESSES(numProcesses), debugInfoRecord(debugInfo(numProcesses)), recoveryMgr(_recoveryMgr) {\r\n VERBOSE DEBUG COUTATOMIC(\"constructor record_manager_single_type\"<initThread(tid);\r\n reclaim->initThread(tid);\r\n// enterQuiescentState(tid);\r\n }\r\n \r\n void deinitThread(const int tid) {\r\n reclaim->deinitThread(tid);\r\n }\r\n \r\n inline void clearCounters() {\r\n debugInfoRecord.clear();\r\n }\r\n\r\n inline static bool shouldHelp() { // FOR DEBUGGING PURPOSES\r\n return Reclaim::shouldHelp();\r\n }\r\n inline bool isProtected(const int tid, record_pointer obj) {\r\n return reclaim->isProtected(tid, obj);\r\n }\r\n // for hazard pointers (and reference counting)\r\n inline bool protect(const int tid, record_pointer obj, CallbackType notRetiredCallback, CallbackArg callbackArg, bool hintMemoryBarrier = true) {\r\n return reclaim->protect(tid, obj, notRetiredCallback, callbackArg, hintMemoryBarrier);\r\n }\r\n inline void unprotect(const int tid, record_pointer obj) {\r\n reclaim->unprotect(tid, obj);\r\n }\r\n // warning: qProtect must be reentrant and lock-free (=== async-signal-safe)\r\n inline bool qProtect(const int tid, record_pointer obj, CallbackType notRetiredCallback, CallbackArg callbackArg, bool hintMemoryBarrier = true) {\r\n return reclaim->qProtect(tid, obj, notRetiredCallback, callbackArg, hintMemoryBarrier);\r\n }\r\n inline void qUnprotectAll(const int tid) {\r\n assert(!Reclaim::supportsCrashRecovery() || isQuiescent(tid));\r\n reclaim->qUnprotectAll(tid);\r\n }\r\n inline bool isQProtected(const int tid, record_pointer obj) {\r\n return reclaim->isQProtected(tid, obj);\r\n }\r\n \r\n inline static bool supportsCrashRecovery() {\r\n return Reclaim::supportsCrashRecovery();\r\n }\r\n inline static bool quiescenceIsPerRecordType() {\r\n return Reclaim::quiescenceIsPerRecordType();\r\n }\r\n inline bool isQuiescent(const int tid) {\r\n return reclaim->isQuiescent(tid);\r\n }\r\n\r\n // for epoch based reclamation\r\n inline void enterQuiescentState(const int tid) {\r\n// VERBOSE DEBUG2 COUTATOMIC(\"record_manager_single_type::enterQuiescentState(tid=\"<enterQuiescentState(tid);\r\n }\r\n inline void leaveQuiescentState(const int tid, void * const * const reclaimers, const int numReclaimers) {\r\n// assert(isQuiescent(tid));\r\n reclaim->leaveQuiescentState(tid, reclaimers, numReclaimers);\r\n }\r\n\r\n // for all schemes except reference counting\r\n inline void retire(const int tid, record_pointer p) {\r\n assert(!Reclaim::supportsCrashRecovery() || isQuiescent(tid));\r\n reclaim->retire(tid, p);\r\n }\r\n \r\n // for algs that retire before the linearization point of a deletion\r\n inline void unretireLast(const int tid) {\r\n assert(!Reclaim::supportsCrashRecovery() || isQuiescent(tid));\r\n reclaim->unretireLast(tid);\r\n }\r\n\r\n // for all schemes\r\n inline record_pointer allocate(const int tid) {\r\n assert(!Reclaim::supportsCrashRecovery() || isQuiescent(tid));\r\n return pool->get(tid);\r\n }\r\n inline void deallocate(const int tid, record_pointer p) {\r\n assert(!Reclaim::supportsCrashRecovery() || isQuiescent(tid));\r\n pool->add(tid, p);\r\n }\r\n\r\n void printStatus(void) {\r\n long long allocated = debugInfoRecord.getTotalAllocated();\r\n long long allocatedBytes = allocated * sizeof(Record);\r\n long long deallocated = debugInfoRecord.getTotalDeallocated();\r\n long long recycled = debugInfoRecord.getTotalFromPool() - allocated;\r\n COUTATOMIC(\"recmgr status for objects of size \"<getSizeString()<getSizeString()<getSizeString().c_str()))<debugPrintStatus(tid);\r\n }\r\n COUTATOMIC(endl);\r\n \r\n// for (int tid=0;tidNUM_PROCESSES;++tid) {\r\n// COUTATOMIC(\"thread \"<debugPrintStatus(tid);\r\n// \r\n// COUTATOMIC(\" \");\r\n// //COUTATOMIC(\"allocated \"<debugPrintStatus(tid);\r\n// COUTATOMIC(\" \");\r\n// pool->debugPrintStatus(tid);\r\n// COUTATOMIC(\" \");\r\n// COUTATOMIC(\"(given=\"<\r\n#endif\r\n\r\n#ifndef VERBOSE\r\n #define VERBOSE if(0)\r\n#endif\r\n \r\n#include \r\n#include \r\n#include \"globals.h\"\r\n#ifdef USE_DEBUGCOUNTERS\r\n #include \"debugcounter.h\"\r\n#endif\r\n\r\n// for crash recovery\r\nstatic pthread_key_t pthreadkey;\r\nstatic struct sigaction ___act;\r\nstatic void *___singleton = NULL;\r\nextern pthread_key_t pthreadkey;\r\nextern struct sigaction ___act;\r\nextern void *___singleton;\r\n\r\nstatic pthread_t registeredThreads[MAX_TID_POW2];\r\nstatic void *errnoThreads[MAX_TID_POW2];\r\nstatic sigjmp_buf *setjmpbuffers;\r\nextern pthread_t registeredThreads[MAX_TID_POW2];\r\nextern void *errnoThreads[MAX_TID_POW2];\r\nextern sigjmp_buf *setjmpbuffers;\r\n\r\n#ifdef USE_DEBUGCOUNTERS\r\nstatic debugCounter countInterrupted(MAX_TID_POW2);\r\nstatic debugCounter countLongjmp(MAX_TID_POW2);\r\nextern debugCounter countInterrupted;\r\nextern debugCounter countLongjmp;\r\n#endif\r\n#define MAX_THREAD_ADDR 10000\r\n\r\n#ifdef CRASH_RECOVERY_USING_SETJMP\r\n#define CHECKPOINT_AND_RUN_UPDATE(tid, finishedbool) \\\r\n if (MasterRecordMgr::supportsCrashRecovery() && sigsetjmp(setjmpbuffers[(tid)], 0)) { \\\r\n recordmgr->enterQuiescentState((tid)); \\\r\n (finishedbool) = recoverAnyAttemptedSCX((tid), -1); \\\r\n recordmgr->recoveryMgr->unblockCrashRecoverySignal(); \\\r\n } else\r\n#define CHECKPOINT_AND_RUN_QUERY(tid) \\\r\n if (MasterRecordMgr::supportsCrashRecovery() && sigsetjmp(setjmpbuffers[(tid)], 0)) { \\\r\n recordmgr->enterQuiescentState((tid)); \\\r\n recordmgr->recoveryMgr->unblockCrashRecoverySignal(); \\\r\n } else\r\n#endif\r\n\r\n// warning: this crash recovery code will only work if you've created a SINGLE instance of bst during an execution.\r\n// there are ways to make it work for multiple instances; i just haven't done that.\r\ntemplate \r\nvoid crashhandler(int signum, siginfo_t *info, void *uctx) {\r\n MasterRecordMgr * const recordmgr = (MasterRecordMgr * const) ___singleton;\r\n#ifdef SIGHANDLER_IDENTIFY_USING_PTHREAD_GETSPECIFIC\r\n int tid = (int) ((long) pthread_getspecific(pthreadkey));\r\n#endif\r\n TRACE COUTATOMICTID(\"received signal \"<isQuiescent(tid)) {\r\n#ifdef PERFORM_RESTART_IN_SIGHANDLER\r\n recordmgr->enterQuiescentState(tid);\r\n #ifdef USE_DEBUGCOUNTERS\r\n DEBUG countLongjmp.inc(tid);\r\n #endif\r\n __sync_synchronize();\r\n #ifdef CRASH_RECOVERY_USING_SETJMP\r\n siglongjmp(setjmpbuffers[tid], 1);\r\n #endif\r\n#endif\r\n }\r\n // otherwise, i simply continue my operation as if nothing happened.\r\n // this lets me behave nicely when it would be dangerous for me to be\r\n // restarted (being in a Q state is analogous to having interrupts \r\n // disabled in an operating system kernel; however, whereas disabling\r\n // interrupts blocks other processes' progress, being in a Q state\r\n // implies that you cannot block the progress of any other thread.)\r\n}\r\n\r\ntemplate \r\nclass RecoveryMgr {\r\npublic:\r\n const int NUM_PROCESSES;\r\n const int neutralizeSignal;\r\n \r\n inline int getTidInefficient(const pthread_t me) {\r\n int tid = -1;\r\n for (int i=0;i; // specify signal handler\r\n ___act.sa_flags = SA_RESTART | SA_SIGINFO; // restart any interrupted sys calls instead of silently failing\r\n sigfillset(&___act.sa_mask); // block signals during handler\r\n if (sigaction(_neutralizeSignal, &___act, NULL)) {\r\n COUTATOMIC(\"ERROR: could not register signal handler for signal \"<<_neutralizeSignal<dtime == TIMESTAMP_NOT_SET) ; true; })\r\n#else\r\n#define WAIT_FOR_DTIME(node) ({ false; })\r\n#endif\r\n\r\n#include \r\n#include \r\n#include \"rq_debugging.h\"\r\n#include \"dcss_plus_impl.h\"\r\n\r\ntemplate \r\ninline bool contains(T ** nullTerminatedArray, T * element) {\r\n for (int i=0;nullTerminatedArray[i];++i) {\r\n if (nullTerminatedArray[i] == element) return true;\r\n }\r\n return false;\r\n}\r\n\r\ntemplate \r\ninline bool contains(T * array, const int numElements, T element) {\r\n for (int i=0;i\r\nclass RQProvider {\r\nprivate:\r\n struct __rq_thread_data {\r\n #define __RQ_THREAD_DATA_SIZE 1024\r\n #define MAX_NODES_DELETED_ATOMICALLY 8\r\n #define CODE_COVERAGE_MAX_PATHS 11\r\n union {\r\n struct { // anonymous struct inside anonymous union means we don't need to type anything special to access these variables\r\n long long rq_lin_time;\r\n HashList * hashlist;\r\n#ifdef COUNT_CODE_PATH_EXECUTIONS\r\n long long codePathExecutions[CODE_COVERAGE_MAX_PATHS];\r\n#endif\r\n volatile char padding0[PREFETCH_SIZE_BYTES];\r\n void * announcements[MAX_NODES_DELETED_ATOMICALLY];\r\n int numAnnouncements;\r\n };\r\n char bytes[__RQ_THREAD_DATA_SIZE]; // avoid false sharing (note: anon struct above contains around 96 bytes)\r\n };\r\n } __attribute__((aligned(__RQ_THREAD_DATA_SIZE)));\r\n \r\n#ifdef COUNT_CODE_PATH_EXECUTIONS\r\n #define COUNT_CODE_PATH(path) { assert((path) < CODE_COVERAGE_MAX_PATHS); (++threadData[tid].codePathExecutions[(path)]); }\r\n long long codePathExecutions[CODE_COVERAGE_MAX_PATHS];\r\n#else\r\n #define COUNT_CODE_PATH(path) \r\n#endif\r\n\r\n #define TIMESTAMP_NOT_SET 0\r\n #define HASHLIST_INIT_CAPACITY_POW2 (1<<8)\r\n\r\n const int NUM_PROCESSES;\r\n volatile char padding0[PREFETCH_SIZE_BYTES];\r\n volatile long long timestamp = 1;\r\n volatile char padding1[PREFETCH_SIZE_BYTES];\r\n __rq_thread_data * threadData;\r\n\r\n #define NODE_DELETED_BEFORE_RQ 0\r\n #define NODE_DELETED_AFTER_RQ 1\r\n #define NODE_NOT_DELETED_BY_THREAD -1\r\n dcsspProvider * prov;\r\n \r\n DataStructure * ds;\r\n RecordManager * const recmgr;\r\n\r\n int init[MAX_TID_POW2] = {0,};\r\n\r\npublic:\r\n RQProvider(const int numProcesses, DataStructure * ds, RecordManager * recmgr) : NUM_PROCESSES(numProcesses), ds(ds), recmgr(recmgr) {\r\n prov = new dcsspProvider(numProcesses);\r\n threadData = new __rq_thread_data[numProcesses];\r\n DEBUG_INIT_RQPROVIDER(numProcesses);\r\n#ifdef COUNT_CODE_PATH_EXECUTIONS\r\n for (int i=0;ideinitThread(tid);\r\n// threadData[tid].hashlist->destroy();\r\n// delete threadData[tid].hashlist;\r\n// }\r\n \r\n prov->debugPrint();\r\n delete prov;\r\n delete[] threadData;\r\n DEBUG_DEINIT_RQPROVIDER(NUM_PROCESSES);\r\n }\r\n \r\n // invoke before a given thread can invoke any functions on this object\r\n void initThread(const int tid) {\r\n if (init[tid]) return; else init[tid] = !init[tid];\r\n\r\n prov->initThread(tid);\r\n threadData[tid].hashlist = new HashList();\r\n threadData[tid].hashlist->init(HASHLIST_INIT_CAPACITY_POW2);\r\n threadData[tid].numAnnouncements = 0;\r\n for (int i=0;ideinitThread(tid);\r\n threadData[tid].hashlist->destroy();\r\n delete threadData[tid].hashlist;\r\n#ifdef COUNT_CODE_PATH_EXECUTIONS\r\n for (int i=0;iitime = TIMESTAMP_NOT_SET;\r\n node->dtime = TIMESTAMP_NOT_SET;\r\n }\r\n\r\n // for each address addr that is modified by rq_linearize_update_at_write\r\n // or rq_linearize_update_at_cas, you must replace any initialization of addr\r\n // with invocations of rq_write_addr\r\n //\r\n // NOTE: this CANNOT be used on fields that might be concurrently being modified\r\n // by an invocation of rq_linearize_update_at_write or\r\n // rq_linearize_update_at_cas\r\n template \r\n inline void write_addr(const int tid, T volatile * const addr, const T val) {\r\n if (is_pointer::value) {\r\n prov->writePtr((casword_t *) addr, (casword_t) val);\r\n } else {\r\n prov->writeVal((casword_t *) addr, (casword_t) val);\r\n }\r\n }\r\n\r\n // for each address addr that is modified by rq_linearize_update_at_write\r\n // or rq_linearize_update_at_cas, you must replace any reads of addr with\r\n // invocations of rq_read_addr\r\n template \r\n inline T read_addr(const int tid, T volatile * const addr) {\r\n return (T) ((is_pointer::value)\r\n ? prov->readPtr(tid, (casword_t *) addr)\r\n : prov->readVal(tid, (casword_t *) addr));\r\n }\r\n \r\n // IF DATA STRUCTURE PERFORMS LOGICAL DELETION\r\n // run some time BEFORE the physical deletion of a node\r\n // whose key has ALREADY been logically deleted.\r\n void announce_physical_deletion(const int tid, NodeType * const * const deletedNodes) {\r\n int i;\r\n for (i=0;deletedNodes[i];++i) {\r\n threadData[tid].announcements[threadData[tid].numAnnouncements+i] = deletedNodes[i];\r\n }\r\n SOFTWARE_BARRIER;\r\n threadData[tid].numAnnouncements += i;\r\n assert(threadData[tid].numAnnouncements <= MAX_NODES_DELETED_ATOMICALLY);\r\n SOFTWARE_BARRIER;\r\n }\r\n\r\n // IF DATA STRUCTURE PERFORMS LOGICAL DELETION\r\n // run AFTER performing announce_physical_deletion,\r\n // if the cas that was trying to physically delete node failed.\r\n void physical_deletion_failed(const int tid, NodeType * const * const deletedNodes) {\r\n for (int i=0;deletedNodes[i];++i) {\r\n --threadData[tid].numAnnouncements;\r\n }\r\n assert(threadData[tid].numAnnouncements >= 0);\r\n }\r\n \r\n // IF DATA STRUCTURE PERFORMS LOGICAL DELETION\r\n // run AFTER performing announce_physical_deletion,\r\n // if the cas that was trying to physically delete node succeeded.\r\n void physical_deletion_succeeded(const int tid, NodeType * const * const deletedNodes) {\r\n int i;\r\n for (i=0;deletedNodes[i];++i) {\r\n recmgr->retire(tid, deletedNodes[i]);\r\n }\r\n SOFTWARE_BARRIER; // ensure nodes are placed in the epoch bag BEFORE they are removed from announcements.\r\n threadData[tid].numAnnouncements -= i;\r\n assert(threadData[tid].numAnnouncements >= 0);\r\n }\r\n \r\nprivate:\r\n \r\n inline void set_insertion_timestamps(\r\n const int tid,\r\n const long long ts,\r\n NodeType * const * const insertedNodes,\r\n NodeType * const * const deletedNodes) {\r\n \r\n // set insertion timestamps\r\n // for each i_node in insertedNodes\r\n for (int i_nodeix=0;insertedNodes[i_nodeix];++i_nodeix) {\r\n insertedNodes[i_nodeix]->itime = ts;\r\n }\r\n }\r\n\r\n inline void set_deletion_timestamps(\r\n const int tid,\r\n const long long ts,\r\n NodeType * const * const insertedNodes,\r\n NodeType * const * const deletedNodes) {\r\n \r\n // set deletion timestamps\r\n // for each d_node in deletedNodes\r\n for (int d_nodeix=0;deletedNodes[d_nodeix];++d_nodeix) {\r\n deletedNodes[d_nodeix]->dtime = ts;\r\n }\r\n }\r\n \r\npublic:\r\n \r\n // replace the linearization point of an update that inserts or deletes nodes\r\n // with an invocation of this function if the linearization point is a WRITE\r\n template \r\n inline T linearize_update_at_write(\r\n const int tid,\r\n T volatile * const lin_addr,\r\n const T& lin_newval,\r\n NodeType * const * const insertedNodes,\r\n NodeType * const * const deletedNodes) {\r\n\r\n if (!logicalDeletion) {\r\n // physical deletion will happen at the same time as logical deletion\r\n announce_physical_deletion(tid, deletedNodes);\r\n }\r\n \r\n casword_t old1;\r\n while (true) {\r\n old1 = (casword_t) timestamp;\r\n\r\n casword_t old2 = (is_pointer::value)\r\n ? (casword_t) prov->readPtr(tid, (casword_t *) lin_addr)\r\n : (casword_t) prov->readVal(tid, (casword_t *) lin_addr);\r\n casword_t new2 = (casword_t) lin_newval;\r\n dcsspresult_t result = (is_pointer::value)\r\n ? prov->dcsspPtr(tid, (casword_t *) ×tamp, old1, (casword_t *) lin_addr, old2, new2, (void **) insertedNodes, (void **) deletedNodes)\r\n : prov->dcsspVal(tid, (casword_t *) ×tamp, old1, (casword_t *) lin_addr, old2, new2, (void **) insertedNodes, (void **) deletedNodes);\r\n if (result.status == DCSSP_SUCCESS) {\r\n break;\r\n }\r\n }\r\n //DELAY_UP_TO(10000);\r\n\r\n set_insertion_timestamps(tid, old1 /* timestamp */, insertedNodes, deletedNodes);\r\n set_deletion_timestamps(tid, old1 /* timestamp */, insertedNodes, deletedNodes);\r\n \r\n // discard the payloads (insertedNodes and deletedNodes) in this thread's descriptor\r\n // so other threads can't access them far in the future if we become QUIESCENT and sleep for a long time\r\n // (must be performed after setting itimes and dtimes, but before enterQuiescentState)\r\n prov->discardPayloads(tid);\r\n\r\n if (!logicalDeletion) {\r\n // physical deletion will happen at the same time as logical deletion\r\n physical_deletion_succeeded(tid, deletedNodes);\r\n }\r\n \r\n#if defined USE_RQ_DEBUGGING\r\n DEBUG_RECORD_UPDATE_CHECKSUM(tid, old1 /* timestamp */, insertedNodes, deletedNodes, ds);\r\n#endif\r\n return lin_newval;\r\n }\r\n\r\n // replace the linearization point of an update that inserts or deletes nodes\r\n // with an invocation of this function if the linearization point is a CAS\r\n template \r\n inline T linearize_update_at_cas(\r\n const int tid,\r\n T volatile * const lin_addr,\r\n const T& lin_oldval,\r\n const T& lin_newval,\r\n NodeType * const * const insertedNodes,\r\n NodeType * const * const deletedNodes) {\r\n\r\n if (!logicalDeletion) {\r\n // physical deletion will happen at the same time as logical deletion\r\n announce_physical_deletion(tid, deletedNodes);\r\n }\r\n \r\n casword_t old2 = (casword_t) lin_oldval;\r\n casword_t new2 = (casword_t) lin_newval;\r\n dcsspresult_t result;\r\n while (true) {\r\n casword_t old1 = (casword_t) timestamp;\r\n\r\n result = (is_pointer::value)\r\n ? prov->dcsspPtr(tid, (casword_t *) ×tamp, old1 /* timestamp */, (casword_t *) lin_addr, old2, new2, (void **) insertedNodes, (void **) deletedNodes)\r\n : prov->dcsspVal(tid, (casword_t *) ×tamp, old1 /* timestamp */, (casword_t *) lin_addr, old2, new2, (void **) insertedNodes, (void **) deletedNodes);\r\n if (result.status == DCSSP_SUCCESS) {\r\n //DELAY_UP_TO(1000);\r\n\r\n set_insertion_timestamps(tid, old1 /* timestamp */, insertedNodes, deletedNodes);\r\n set_deletion_timestamps(tid, old1 /* timestamp */, insertedNodes, deletedNodes);\r\n\r\n // discard the payloads (insertedNodes and deletedNodes) in this thread's descriptor\r\n // so other threads can't access them far in the future if we become QUIESCENT and sleep for a long time\r\n // (must be performed after setting itimes and dtimes, but before enterQuiescentState)\r\n prov->discardPayloads(tid);\r\n \r\n if (!logicalDeletion) {\r\n // physical deletion will happen at the same time as logical deletion\r\n physical_deletion_succeeded(tid, deletedNodes);\r\n } \r\n \r\n#if defined USE_RQ_DEBUGGING\r\n DEBUG_RECORD_UPDATE_CHECKSUM(tid, old1 /* timestamp */, insertedNodes, deletedNodes, ds);\r\n#endif\r\n return lin_oldval;\r\n } else if (result.status == DCSSP_FAILED_ADDR2) {\r\n // failed due to original CAS's failure (NOT due to the timestamp changing)\r\n\r\n if (!logicalDeletion) {\r\n // physical deletion will happen at the same time as logical deletion\r\n physical_deletion_failed(tid, deletedNodes);\r\n }\r\n \r\n break;\r\n }\r\n }\r\n assert(result.status == DCSSP_FAILED_ADDR2);\r\n assert(old2 != result.failed_val);\r\n return (T) result.failed_val;\r\n }\r\n\r\n // invoke at the start of each traversal\r\n inline void traversal_start(const int tid) {\r\n threadData[tid].hashlist->clear();\r\n threadData[tid].rq_lin_time = __sync_add_and_fetch(×tamp, 1); // linearize rq here!\r\n }\r\n\r\nprivate:\r\n // invoke each time a traversal visits a node with a key in the desired range:\r\n // if the node belongs in the range query, it will be placed in rqResult[index]\r\n inline int __traversal_try_add(const int tid, NodeType * const node, K * const outputKeys, V * const outputValues, const K& lo, const K& hi, bool foundDuringTraversal) {\r\n\r\n // rqResultKeys should have space for MAX_KEYS_PER_NODE keys, AT LEAST\r\n \r\n // in the following, rather than having deeply nested if-else blocks,\r\n // we return asap, and list facts that must be true if we didn't return\r\n assert(foundDuringTraversal || !logicalDeletion || ds->isLogicallyDeleted(tid, node));\r\n\r\n // TODO: ensure this makes sense when called with announced nodes\r\n \r\n long long itime = node->itime;\r\n if (itime != TIMESTAMP_NOT_SET & node->itime >= threadData[tid].rq_lin_time) return 0; // node was inserted after the range query\r\n // fact: either itime was not set above, or node was inserted before rq\r\n \r\n ///////////////////////// HANDLE UNKNOWN ITIME /////////////////////////\r\n\r\n // TODO: try adding a bit of spinning before falling back to the full lock-free solution\r\n \r\n // determine if any other process inserted, or is trying to insert node, and, if so, when\r\n for (int otherTid=0; (itime = node->itime) == TIMESTAMP_NOT_SET && otherTidgetDescriptorTagptr(otherTid); // try to get a snapshot of otherTid's dcssp descriptor\r\n dcsspdesc_t snap;\r\n if (!prov->getDescriptorSnapshot(tagptr, &snap)) { // we failed to obtain a snapshot, which means that while getDescriptorSnapshot() was running, the process finished one dcssp, and started a new dcssp.\r\n continue; // goto check next process // if the finished dcssp inserted node, then before the next dcssp by the same process, node->itime is set. so, we check whether itime is set.\r\n }\r\n // fact: we obtained a snapshot\r\n\r\n if (!SNAPSHOT_CONTAINS_INSERTED_NODE(snap, node)) continue; // goto check next process\r\n // fact: otherTid is trying/tried to insert node\r\n\r\n int state = MUTABLES_UNPACK_FIELD(snap.mutables, DCSSP_MUTABLES_MASK_STATE, DCSSP_MUTABLES_OFFSET_STATE);\r\n if (state == DCSSP_STATE_FAILED) { // the operation described by snap did not insert node, so either this process did not insert it, or the process inserted/inserted it in a PREVIOUS operation, so it must have already set itime as appropriate\r\n continue; // goto check next process\r\n } else if (state == DCSSP_STATE_SUCCEEDED) { // the dcssp operation finished, and inserted node. to determine WHEN it inserted node, we look at the argument old1 to the dcssp, which contains the timestamp when the dcssp took place. (observe that this is the value process otherTid would write to node->itime)\r\n if (snap.old1 >= threadData[tid].rq_lin_time) return 0; // node was inserted after rq\r\n break; // process inserted node at time snap.old1, BEFORE the RQ\r\n }\r\n // fact: state is UNDECIDED\r\n \r\n // now we try to help\r\n casword_t addr2 = *snap.addr2;\r\n if (addr2 == tagptr) { // addr2 indeed points to the dcssp descriptor. the linearization point of the dcssp operation occurs after this step, so the dcssp might have been linearized, but not yet had its state set.\r\n prov->helpProcess(tid, otherTid); // we need to know what its final state will be to determine whether it successfully inserted node. so, we HELP otherTid finish its dcssp.\r\n }\r\n // note: the following all happens in BOTH the cases where addr2 == tagptr and where addr2 != tagptr, except there is some extra work if addr2 != tagptr and state2 != SUCCEEDED. i've folded the two cases together simply for compactness / less repetition.\r\n\r\n // then, we reread the state\r\n bool valid = false;\r\n dcsspdesc_t * ptr = prov->getDescriptorPtr(tagptr);\r\n int state2 = DESC_READ_FIELD(valid, ptr->mutables, tagptr, DCSSP_MUTABLES_MASK_STATE, DCSSP_MUTABLES_OFFSET_STATE);\r\n if (!valid) continue; // goto check next process // the read of the state field was invalid, which means that the dcssp operation has terminated, the next dcssp operation by otherTid has begun. since the next dcssp operation has begun, and each high-level data structure operation performs only one successful dcssp (in a call to linearize_update_at_...), if this dcssp that finished in fact inserted node, then the next dcssp would be part of the next high-level operation. thus, if node->itime was inserted by the finished dcssp, then the high-level operation that performed this dcssp will already have set node->itime.\r\n // fact: the read of state was valid\r\n \r\n if (state2 == DCSSP_STATE_SUCCEEDED) { // we are in case (b) described above. the dcssp operation finished, and insert node. to determine WHEN it inserted node, we look at the argument old1 to the dcssp, which contains the timestamp when the dcssp took place. (observe that this is the value process otherTid would write to node->itime)\r\n if (snap.old1 >= threadData[tid].rq_lin_time) return 0; // node was inserted after rq\r\n break; // process inserted node at time snap.old1, BEFORE the RQ\r\n } else { // undecided or failed\r\n continue; // goto check next process // we are in case (a) or case (c) described above. so, otherTid did NOT delete node.\r\n }\r\n }\r\n if (itime != TIMESTAMP_NOT_SET && itime >= threadData[tid].rq_lin_time) return 0; // node was inserted after rq\r\n\r\n /////////////// HANDLE LOGICAL DELETION AND CHECK DTIME ////////////////\r\n \r\n long long dtime = TIMESTAMP_NOT_SET;\r\n\r\n if (!logicalDeletion && foundDuringTraversal) goto tryAddToRQ; // no logical deletion. since node was inserted before the range query, and the traversal encountered it, it must have been deleted AFTER the traversal encountered it.\r\n // fact: no logical deletion ==> did not find node during traversal\r\n\r\n dtime = node->dtime;\r\n if (dtime != TIMESTAMP_NOT_SET && dtime < threadData[tid].rq_lin_time) return 0;\r\n // fact: either dtime was not set above, or node was deleted after rq\r\n\r\n if (logicalDeletion && !ds->isLogicallyDeleted(tid, node)) goto tryAddToRQ; // if logical deletion is used with marking, the fact that node was inserted before the range query, and that the traversal encountered node, is NOT enough to argue that node was in the data structure when the traversal started. why? when the traversal encountered node, it might have already been marked. so, we check if node is marked. if not, then the node has not yet been deleted.\r\n // fact: logical deletion ==> node has been logically deleted\r\n \r\n ///////////////////////// HANDLE UNKNOWN DTIME /////////////////////////\r\n \r\n // determine if any other process is trying/tried to delete node\r\n for (int otherTid=0; (dtime = node->dtime) == TIMESTAMP_NOT_SET && otherTidgetDescriptorTagptr(otherTid); // try to get a snapshot of otherTid's dcssp descriptor\r\n dcsspdesc_t snap;\r\n if (!prov->getDescriptorSnapshot(tagptr, &snap)) { // we failed to obtain a snapshot, which means that while getDescriptorSnapshot() was running, the process finished one dcssp, and started a new dcssp.\r\n continue; // goto check next process // if the finished dcssp deleted node, then before the next dcssp by the same process, node->dtime is set (and it will be seen after the loop).\r\n }\r\n // fact: we obtained a snapshot\r\n\r\n if (!SNAPSHOT_CONTAINS_DELETED_NODE(snap, node)) continue; // goto check next process\r\n // fact: otherTid is trying/tried to delete node\r\n\r\n // we must determine whether otherTid's dcssp operation (whose descriptor we obtained a snapshot of) has been linearized, and whether it was successful.\r\n // we use the following facts.\r\n // (1) the dcssp descriptor has a state that is initially UNDECIDED, and becomes SUCCEEDED or FAILED after the dcssp has been linearized.\r\n // (2) a dcssp that succeeds or fails changes *snap.addr2 to tagptr, then reads *snap.addr1 and linearizes, then sets its state to SUCCEEDED or FAILED, then changes *snap.addr2 from tagptr to another value.\r\n // (3) once *snap.addr2 has been changed from tagptr to another value, it can never again contain tagptr.\r\n // (4) each high-level data structure operation invokes dcssp only via linearize_update_at_write or linearize_update_at_cas, and only performs one invocation of linearize_update_at_write or one /successful/ invocation of linearize_update_at_cas (and possibly many unsuccessful invocations of linearize_update_at_cas).\r\n // (5) if a dcssp operation deletes node, then before the next dcssp by the same process, node->dtime is set.\r\n // so, we check the state of the dcssp operation. if it is SUCCEEDED or FAILED, we have our answer. but, if the state is UNDECIDED, the dcssp may or may not have been linearized.\r\n // in the latter case, to determine whether it has been linearized, we would like to HELP the dcssp operation to complete.\r\n // note, however, that the help procedure for the dcssp algorithm can be invoked only if otherTid has already changed *snap.addr2 to tagptr.\r\n // thus, we must determine whether otherTid has changed *snap.addr2 to tagptr, before we can help the dcssp operation.\r\n // so, we read *snap.addr2. if we see that it contains tagptr, then we can help the dcssp.\r\n // otherwise, one of the following must be true:\r\n // (a) otherTid has not yet changed *snap.addr2 to tagptr, or\r\n // (b) otherTid changed *snap.addr2 to tagptr, then it (or a helper) changed its state to SUCCEEDED, then changed *snap.addr2 to a different value (never again to contain tagptr), or\r\n // (c) otherTid changed *snap.addr2 to tagptr, then it (or a helper) changed its state to FAILED, then changed *snap.addr2 to a different value (never again to contain tagptr).\r\n // in case (a), we know that the dcssp has not yet been linearized.\r\n // in case (b), the dcssp has been linearized, has state SUCCEEDED, and deleted node.\r\n // in case (c), the dcssp has been linearized, has state FAILED, and did NOT delete node.\r\n // so, after reading *snap.addr2, we read the dcssp operation's state again to determine which case has occurred.\r\n\r\n int state = MUTABLES_UNPACK_FIELD(snap.mutables, DCSSP_MUTABLES_MASK_STATE, DCSSP_MUTABLES_OFFSET_STATE);\r\n if (state == DCSSP_STATE_FAILED) { // the operation described by snap did not insert/delete node, so either this process did not insert/delete it, or the process inserted/deleted it in a PREVIOUS operation, so it must have already set itime/dtime as appropriate\r\n continue; // goto check next process\r\n } else if (state == DCSSP_STATE_SUCCEEDED) { // the dcssp operation finished, and deleted node. to determine WHEN it deleted node, we look at the argument old1 to the dcssp, which contains the timestamp when the dcssp took place. (observe that this is the value process otherTid would write to node->dtime)\r\n if (WAIT_FOR_DTIME(node)) {\r\n // the following assertions are thread safe ONLY if WAIT_FOR_DTIME actually waits! (which is true only if it returns true, which is true only if RQ_LOCKFREE_WAITS_FOR_DTIME is defined)\r\n assert(snap.old1 <= node->dtime);\r\n assert((snap.old1 >= threadData[tid].rq_lin_time) == (node->dtime >= threadData[tid].rq_lin_time));\r\n assert(foundDuringTraversal || node->dtime == snap.old1);\r\n assert(!foundDuringTraversal || node->dtime == snap.old1);\r\n }\r\n if (snap.old1 < threadData[tid].rq_lin_time) return 0; // node was deleted before rq\r\n goto tryAddToRQ; // node was deleted by this process after rq\r\n }\r\n // fact: state is UNDECIDED\r\n \r\n // maya: in logicalDeletion, since the node is marked the DCSS was successful and only the dtime is not yet set. Thus, UNDECIDED means that otherThread did not mark the node, it was some other thread and we can continue. \r\n if (logicalDeletion) continue; // goto check next process\r\n\r\n // now we try to help\r\n casword_t addr2 = *snap.addr2;\r\n if (addr2 == tagptr) { // TODO: prove it is impossible to execute this block with logical deletion (idea: since node is marked, either (1) otherTid marked it earlier with a DCSS whose state is SUCCEEDED, or (2) someone else marked node, so otherTid cannot successfully CAS addr2.)\r\n // addr2 indeed points to the dcssp descriptor. the linearization point of the dcssp operation occurs after this step, so the dcssp might have been linearized, but not yet had its state set.\r\n prov->helpProcess(tid, otherTid); // we need to know what its final state will be to determine whether it successfully deleted node. so, we HELP otherTid finish its dcssp.\r\n }\r\n // note: the following all happens in BOTH the cases where addr2 == tagptr and where addr2 != tagptr, except there is some extra work if addr2 != tagptr and state2 != SUCCEEDED. i've folded the two cases together simply for compactness / less repetition.\r\n\r\n // then, we reread the state\r\n bool valid = false;\r\n dcsspdesc_t * ptr = prov->getDescriptorPtr(tagptr);\r\n int state2 = DESC_READ_FIELD(valid, ptr->mutables, tagptr, DCSSP_MUTABLES_MASK_STATE, DCSSP_MUTABLES_OFFSET_STATE);\r\n if (!valid) continue; // goto check next process // the read of the state field was invalid, which means that the dcssp operation has terminated, the next dcssp operation by otherTid has begun. since the next dcssp operation has begun, and each high-level data structure operation performs only one successful dcssp (in a call to linearize_update_at_...), if this dcssp that finished in fact deleted node, then the next dcssp would be part of the next high-level operation. thus, if node->itime was inserted by the finished dcssp, then the high-level operation that performed this dcssp will already have set node->itime.\r\n // fact: the read of state was valid\r\n\r\n if (state2 == DCSSP_STATE_SUCCEEDED) { // we are in case (b) described above. the dcssp operation finished, and deleted node. to determine WHEN it deleted node, we look at the argument old1 to the dcssp, which contains the timestamp when the dcssp took place. (observe that this is the value process otherTid would write to node->dtime)\r\n if (snap.old1 >= threadData[tid].rq_lin_time) goto tryAddToRQ;\r\n return 0; // do not add to rq\r\n } else { // undecided or failed\r\n continue; // goto check next process // we are in case (a) or case (c) described above. so, otherTid did NOT delete node.\r\n }\r\n }\r\n if(dtime == TIMESTAMP_NOT_SET) {\r\n assert(!logicalDeletion); \r\n assert(!foundDuringTraversal);\r\n goto tryAddToRQ; // no process deleted node before the range query\r\n }\r\n COUNT_CODE_PATH(9);\r\n if (dtime >= threadData[tid].rq_lin_time) goto tryAddToRQ;\r\n return 0; // do not add to rq\r\n\r\n ///////////////////// TRY TO ADD NODE'S KEYS TO RQ /////////////////////\r\n // note: this way of organizing this decision tree favors trees with fat multi-key nodes, because getKeys is delayed as long as possible.\r\n \r\ntryAddToRQ:\r\n // fetch the node's keys that are in the set\r\n int cnt = ds->getKeys(tid, node, outputKeys, outputValues);\r\n assert(cnt < RQ_DEBUGGING_MAX_KEYS_PER_NODE);\r\n if (cnt == 0) return 0; // node doesn't contain any keys that are in the set and in the desired range\r\n \r\n // note: in the following loop, we shift keys in the outputKeys array left to eliminate any that ultimately should not be added to the range query\r\n int numNewKeys = 0;\r\n for (int i=0;iisInRange(outputKeys[i], lo, hi)) goto doNotAddToRQ; // key is NOT in the desired range\r\n if (threadData[tid].hashlist->contains(outputKeys[i])) goto doNotAddToRQ; // key is already in the range query\r\n outputKeys[numNewKeys] = outputKeys[i]; // save this as a new key added to the RQ\r\n outputValues[numNewKeys] = outputValues[i];\r\n ++numNewKeys;\r\n\r\ndoNotAddToRQ: (0);\r\n }\r\n return numNewKeys;\r\n }\r\n\r\n inline void traversal_try_add(const int tid, NodeType * const node, K * const rqResultKeys, V * const rqResultValues, int * const startIndex, const K& lo, const K& hi, bool foundDuringTraversal) {\r\n//#if defined MICROBENCH && !defined NDEBUG\r\n// assert(*startIndex < RQSIZE); // note: this assert is a hack. it should be *startIndex < size of rqResultKeys\r\n// if (*startIndex > RQSIZE) {\r\n// cout<<\"ERROR: *startIndex=\"<<(*startIndex)<<\" is unexpectedly greater than or equal to RQSIZE=\"<insert(rqResultKeys[(*startIndex)++]);\r\n }\r\n // note: the above increments startIndex\r\n#if defined MICROBENCH\r\n assert(*startIndex <= RQSIZE);\r\n#endif\r\n }\r\n \r\npublic:\r\n inline void traversal_try_add(const int tid, NodeType * const node, K * const rqResultKeys, V * const rqResultValues, int * const startIndex, const K& lo, const K& hi) {\r\n traversal_try_add(tid, node, rqResultKeys, rqResultValues, startIndex, lo, hi, true);\r\n }\r\n \r\n // invoke at the end of each traversal:\r\n // any nodes that were deleted during the traversal,\r\n // and were consequently missed during the traversal,\r\n // are placed in rqResult[index]\r\n void traversal_end(const int tid, K * const rqResultKeys, V * const rqResultValues, int * const startIndex, const K& lo, const K& hi) {\r\n // todo: possibly optimize by skipping entire blocks if there are many keys to skip (does not seem to be justifiable for 4 work threads and 4 range query threads)\r\n\r\n SOFTWARE_BARRIER;\r\n long long end_timestamp = timestamp;\r\n SOFTWARE_BARRIER;\r\n \r\n#if 0\r\n vector nodes;\r\n \r\n // collect nodes announced by other processes\r\n for (int otherTid=0;otherTid * all_bags[NUM_PROCESSES*NUMBER_OF_EPOCH_BAGS+1];\r\n vector> all_iterators;\r\n int numIterators = 0;\r\n for (int otherTid=0;otherTid * thread_bags[NUMBER_OF_EPOCH_BAGS+1];\r\n recmgr->get((NodeType *) NULL)->reclaim->getSafeBlockbags(otherTid, thread_bags);\r\n for (int i=0;thread_bags[i];++i) {\r\n all_bags[numIterators] = thread_bags[i];\r\n all_iterators.push_back(thread_bags[i]->begin());\r\n ++numIterators;\r\n }\r\n }\r\n\r\n // collect nodes in epoch bags\r\n int numVisitedInEpochBags = 0;\r\n for (int ix = 0; ix < numIterators; ++ix) {\r\n for (; all_iterators[ix] != all_bags[ix]->end(); all_iterators[ix]++) {\r\n NodeType * node = (*all_iterators[ix]);\r\n nodes.push_back(node);\r\n \r\n ++numVisitedInEpochBags;\r\n \r\n long long dtime = node->dtime;\r\n if (dtime != TIMESTAMP_NOT_SET && dtime > end_timestamp) continue;\r\n \r\n if (!(logicalDeletion && canRetireNodesLogicallyDeletedByOtherProcesses)) {\r\n // if we cannot retire nodes that are logically deleted\r\n // by other processes, then we always retire nodes in\r\n // order of increasing dtime values.\r\n // so, the blockbag will be ordered, which means that,\r\n // if dtime is before the RQ, then all remaining nodes\r\n // in this bag were deleted before the RQ.\r\n // so, in this case, we skip to the next bag.\r\n if (dtime != TIMESTAMP_NOT_SET && dtime < threadData[tid].rq_lin_time) break;\r\n }\r\n }\r\n }\r\n \r\n // visit collected nodes\r\n for (auto it = nodes.begin(); it != nodes.end(); it++) {\r\n NodeType * node = *it;\r\n traversal_try_add(tid, node, rqResultKeys, startIndex, lo, hi, false);\r\n }\r\n#else\r\n // collect nodes announced by other processes\r\n for (int otherTid=0;otherTid * all_bags[NUM_PROCESSES*NUMBER_OF_EPOCH_BAGS+1];\r\n vector> all_iterators;\r\n int numIterators = 0;\r\n for (int otherTid=0;otherTid * thread_bags[NUMBER_OF_EPOCH_BAGS+1];\r\n recmgr->get((NodeType *) NULL)->reclaim->getSafeBlockbags(otherTid, thread_bags);\r\n for (int i=0;thread_bags[i];++i) {\r\n all_bags[numIterators] = thread_bags[i];\r\n all_iterators.push_back(thread_bags[i]->begin());\r\n ++numIterators;\r\n }\r\n }\r\n \r\n int numSkippedInEpochBags = 0;\r\n int numVisitedInEpochBags = 0;\r\n for (int ix = 0; ix < numIterators; ++ix) {\r\n for (; all_iterators[ix] != all_bags[ix]->end(); all_iterators[ix]++) {\r\n NodeType * node = (*all_iterators[ix]);\r\n assert(node);\r\n\r\n ++numVisitedInEpochBags;\r\n ++numSkippedInEpochBags;\r\n\r\n long long dtime = node->dtime;\r\n if (dtime != TIMESTAMP_NOT_SET && dtime > end_timestamp) continue;\r\n \r\n --numSkippedInEpochBags;\r\n\r\n if (!(logicalDeletion && canRetireNodesLogicallyDeletedByOtherProcesses)) {\r\n // if we cannot retire nodes that are logically deleted\r\n // by other processes, then we always retire nodes in\r\n // order of increasing dtime values.\r\n // so, the blockbag will be ordered, which means that,\r\n // if dtime is before the RQ, then all remaining nodes\r\n // in this bag were deleted before the RQ.\r\n // so, in this case, we skip to the next bag.\r\n if (dtime != TIMESTAMP_NOT_SET && dtime < threadData[tid].rq_lin_time) break;\r\n }\r\n\r\n traversal_try_add(tid, node, rqResultKeys, rqResultValues, startIndex, lo, hi, false);\r\n }\r\n }\r\n \r\n#endif\r\n \r\n#if defined MICROBENCH && !defined NDEBUG\r\n if (*startIndex > RQSIZE) {\r\n cout<<\"ERROR: *startIndex=\"<<(*startIndex)<<\" is unexpectedly greater than or equal to RQSIZE=\"<\r\n #include \r\n #include \r\n using namespace std;\r\n \r\n #define MAX_NUM_RQ_IN_EXECUTION (1<<20)\r\n\r\n #ifdef RQ_VISITED_IN_BAGS_HISTOGRAM\r\n #include \r\n string twoDigits(int x) {\r\n stringstream ss;\r\n if (x >= 0 && x < 10) {\r\n ss<<\"0\";\r\n }\r\n ss<\r\n void printLogarithmicHistogram(T * valuesOverTime, int numValues) {\r\n constexpr int numBits = sizeof(T)*8;\r\n int histogram[numBits+1];\r\n memset(histogram, 0, sizeof(histogram));\r\n T sum = 0;\r\n int cntNonZero = 0;\r\n for (int i=0;i 1) {\r\n v >>= 1;\r\n ++pow2;\r\n }\r\n assert(pow2 <= numBits);\r\n ++histogram[pow2];\r\n }\r\n for (int i=0;i<=numBits;++i) {\r\n if (histogram[i] > 0) {\r\n cout<<\" (2^\"<\r\n #include \"errors.h\"\r\n #define NO_RQ_CHECKSUM (0)\r\n long long ** threadUpdateChecksum; //[MAX_TID_POW2][MAX_NUM_RQ_IN_EXECUTION];\r\n long long ** threadRQChecksum; //[MAX_TID_POW2][MAX_NUM_RQ_IN_EXECUTION];\r\n #endif\r\n\r\n #ifdef RQ_HISTOGRAM\r\n #include \r\n #define CSV_OUTPUT_FILE \"data.csv\"\r\n std::ofstream ofs;\r\n\r\n #define MAX_RQ_SIZE (1<<16)\r\n __thread int numRQs[MAX_RQ_SIZE+1];\r\n int totalNumRQs[MAX_RQ_SIZE+1];\r\n #endif\r\n\r\n inline void DEBUG_RECORD_RQ_VISITED(const int tid, const long long ts, const int numVisited) {\r\n #ifdef RQ_VISITED_IN_BAGS_HISTOGRAM\r\n if (ts >= MAX_NUM_RQ_IN_EXECUTION) return;\r\n threadNumNodesVisitedInBags[tid][ts] = numVisited;\r\n #endif\r\n }\r\n\r\n inline void DEBUG_RECORD_RQ_SIZE(const int size) {\r\n #ifdef RQ_HISTOGRAM\r\n ++numRQs[size];\r\n #endif\r\n }\r\n\r\n template \r\n inline void DEBUG_RECORD_UPDATE_CHECKSUM(const int tid, const long long timestamp, Node * const * const insertedNodes, Node * const * const deletedNodes, DataStructure * const ds) {\r\n #ifdef RQ_VALIDATION\r\n if (timestamp >= MAX_NUM_RQ_IN_EXECUTION) {\r\n return;\r\n // cout << \"timestamp is: \" << timestamp << endl;\r\n // error(\"timestamp > MAX_NUM_RQ_IN_EXECUTION\");\r\n }\r\n for (int i=0;insertedNodes[i];++i) {\r\n K outputKeys[RQ_DEBUGGING_MAX_KEYS_PER_NODE];\r\n V outputValues[RQ_DEBUGGING_MAX_KEYS_PER_NODE];\r\n int cnt = ds->getKeys(tid, insertedNodes[i], outputKeys, outputValues);\r\n assert(cnt <= RQ_DEBUGGING_MAX_KEYS_PER_NODE);\r\n for (int j=0;jgetKeys(tid, deletedNodes[i], outputKeys, outputValues);\r\n assert(cnt <= RQ_DEBUGGING_MAX_KEYS_PER_NODE);\r\n for (int j=0;j\r\n inline void DEBUG_RECORD_RQ_CHECKSUM(const int tid, const long long timestamp, K const * const rqResult, const int len) {\r\n #ifdef RQ_VALIDATION\r\n if (timestamp >= MAX_NUM_RQ_IN_EXECUTION) return;\r\n //if (timestamp >= MAX_NUM_RQ_IN_EXECUTION) error(\"timestamp > MAX_NUM_RQ_IN_EXECUTION\");\r\n // compute checksum\r\n long long checksum = 0;\r\n for (int i=0;i 0) {\r\n cout<<\"RQ VALIDATION TOTAL FAILURES: \"<\r\n#include \r\n#include \r\n#include \r\n#include \r\n\r\n#define MAX_HTM_ATTEMPTS 30\r\n\r\n#define dosum(src) ({ \\\r\n long long __sum = 0; \\\r\n for (int __i=0;__i\r\nclass RQProvider {\r\nprivate:\r\n struct __rq_thread_data {\r\n #define __RQ_THREAD_DATA_SIZE 1024\r\n union {\r\n struct { // anonymous struct inside anonymous union means we don't need to type anything special to access these variables\r\n long long rq_lin_time;\r\n HashList * hashlist;\r\n volatile char padding0[PREFETCH_SIZE_BYTES];\r\n void * announcements[MAX_NODES_DELETED_ATOMICALLY+1];\r\n int numAnnouncements;\r\n volatile char padding1[PREFETCH_SIZE_BYTES]; // prevent false sharing between htm debugging stats below and announcements\r\n \r\n // htm debugging stuff\r\n int commitWriter;\r\n int abortWriter;\r\n int commitReader;\r\n int abortReader;\r\n int fallback;\r\n };\r\n char bytes[__RQ_THREAD_DATA_SIZE]; // avoid false sharing\r\n };\r\n } __attribute__((aligned(__RQ_THREAD_DATA_SIZE)));\r\n\r\n #define TIMESTAMP_NOT_SET 0\r\n #define HASHLIST_INIT_CAPACITY_POW2 (1<<8)\r\n\r\n const int NUM_PROCESSES;\r\n volatile char padding0[PREFETCH_SIZE_BYTES];\r\n volatile long long timestamp = 1;\r\n volatile char padding1[PREFETCH_SIZE_BYTES];\r\n RWLock rwlock;\r\n volatile char padding2[PREFETCH_SIZE_BYTES];\r\n __rq_thread_data * threadData;\r\n \r\n DataStructure * ds;\r\n RecordManager * const recmgr;\r\n\r\n int init[MAX_TID_POW2] = {0,};\r\n\r\npublic:\r\n RQProvider(const int numProcesses, DataStructure * ds, RecordManager * recmgr) : NUM_PROCESSES(numProcesses), ds(ds), recmgr(recmgr) {\r\n threadData = new __rq_thread_data[numProcesses];\r\n DEBUG_INIT_RQPROVIDER(numProcesses);\r\n }\r\n\r\n ~RQProvider() {\r\n cout<<\"writer commits : \"<destroy();\r\n// delete threadData[tid].hashlist;\r\n// }\r\n delete[] threadData;\r\n DEBUG_DEINIT_RQPROVIDER(NUM_PROCESSES);\r\n }\r\n\r\n // invoke before a given thread can perform any rq_functions\r\n void initThread(const int tid) {\r\n if (init[tid]) return; else init[tid] = !init[tid];\r\n\r\n threadData[tid].hashlist = new HashList();\r\n threadData[tid].hashlist->init(HASHLIST_INIT_CAPACITY_POW2);\r\n threadData[tid].numAnnouncements = 0;\r\n for (int i=0;idestroy();\r\n delete threadData[tid].hashlist;\r\n DEBUG_DEINIT_THREAD(tid);\r\n }\r\n\r\n // invoke whenever a new node is created/initialized\r\n inline void init_node(const int tid, NodeType * const node) {\r\n node->itime = TIMESTAMP_NOT_SET;\r\n node->dtime = TIMESTAMP_NOT_SET;\r\n }\r\n\r\n // for each address addr that is modified by rq_linearize_update_at_write\r\n // or rq_linearize_update_at_cas, you must replace any initialization of addr\r\n // with invocations of rq_write_addr\r\n template \r\n inline void write_addr(const int tid, T volatile * const addr, const T val) {\r\n *addr = val;\r\n }\r\n\r\n // for each address addr that is modified by rq_linearize_update_at_write\r\n // or rq_linearize_update_at_cas, you must replace any reads of addr with\r\n // invocations of rq_read_addr\r\n template \r\n inline T read_addr(const int tid, T volatile * const addr) {\r\n return *addr;\r\n }\r\n\r\n // IF DATA STRUCTURE PERFORMS LOGICAL DELETION\r\n // run some time BEFORE the physical deletion of a node\r\n // whose key has ALREADY been logically deleted.\r\n void announce_physical_deletion(const int tid, NodeType * const * const deletedNodes) {\r\n int i;\r\n for (i=0;deletedNodes[i];++i) {\r\n threadData[tid].announcements[threadData[tid].numAnnouncements+i] = deletedNodes[i];\r\n }\r\n SOFTWARE_BARRIER;\r\n threadData[tid].numAnnouncements += i;\r\n assert(threadData[tid].numAnnouncements <= MAX_NODES_DELETED_ATOMICALLY);\r\n SOFTWARE_BARRIER;\r\n }\r\n\r\n // IF DATA STRUCTURE PERFORMS LOGICAL DELETION\r\n // run AFTER performing announce_physical_deletion,\r\n // if the cas that was trying to physically delete node failed.\r\n void physical_deletion_failed(const int tid, NodeType * const * const deletedNodes) {\r\n for (int i=0;deletedNodes[i];++i) {\r\n --threadData[tid].numAnnouncements;\r\n }\r\n assert(threadData[tid].numAnnouncements >= 0);\r\n }\r\n \r\n // IF DATA STRUCTURE PERFORMS LOGICAL DELETION\r\n // run AFTER performing announce_physical_deletion,\r\n // if the cas that was trying to physically delete node succeeded.\r\n void physical_deletion_succeeded(const int tid, NodeType * const * const deletedNodes) {\r\n int i;\r\n for (i=0;deletedNodes[i];++i) {\r\n recmgr->retire(tid, deletedNodes[i]);\r\n }\r\n SOFTWARE_BARRIER; // ensure nodes are placed in the epoch bag BEFORE they are removed from announcements.\r\n threadData[tid].numAnnouncements -= i;\r\n assert(threadData[tid].numAnnouncements >= 0);\r\n }\r\n \r\nprivate:\r\n \r\n inline void set_insertion_timestamps(\r\n const int tid,\r\n const long long ts,\r\n NodeType * const * const insertedNodes,\r\n NodeType * const * const deletedNodes) {\r\n \r\n // set insertion timestamps\r\n // for each i_node in insertedNodes\r\n for (int i_nodeix=0;insertedNodes[i_nodeix];++i_nodeix) {\r\n insertedNodes[i_nodeix]->itime = ts;\r\n }\r\n }\r\n\r\n inline void set_deletion_timestamps(\r\n const int tid,\r\n const long long ts,\r\n NodeType * const * const insertedNodes,\r\n NodeType * const * const deletedNodes) {\r\n \r\n // set deletion timestamps\r\n // for each d_node in deletedNodes\r\n for (int d_nodeix=0;deletedNodes[d_nodeix];++d_nodeix) {\r\n deletedNodes[d_nodeix]->dtime = ts;\r\n }\r\n }\r\n \r\npublic:\r\n\r\n // replace the linearization point of an update that inserts or deletes nodes\r\n // with an invocation of this function if the linearization point is a WRITE\r\n template \r\n inline T linearize_update_at_write(\r\n const int tid,\r\n T volatile * const lin_addr,\r\n const T& lin_newval,\r\n NodeType * const * const insertedNodes,\r\n NodeType * const * const deletedNodes) {\r\n\r\n if (!logicalDeletion) {\r\n // physical deletion will happen at the same time as logical deletion\r\n announce_physical_deletion(tid, deletedNodes);\r\n }\r\n\r\n // htm path\r\n long long ts;\r\n int limit = MAX_HTM_ATTEMPTS;\r\n while (limit--) {\r\n while (rwlock.isWriteLocked()) {}\r\n if (XBEGIN() == _XBEGIN_STARTED) {\r\n if (rwlock.isWriteLocked()) XABORT(1);\r\n ts = timestamp;\r\n *lin_addr = lin_newval; // original linearization point\r\n XEND();\r\n ++threadData[tid].commitReader;\r\n goto committed;\r\n } else ++threadData[tid].abortReader;\r\n\r\n }\r\n \r\n // fallback path\r\n ++threadData[tid].fallback;\r\n rwlock.readLock();\r\n ts = timestamp;\r\n *lin_addr = lin_newval; // original linearization point\r\n rwlock.readUnlock();\r\n\r\ncommitted:\r\n set_insertion_timestamps(tid, ts, insertedNodes, deletedNodes);\r\n set_deletion_timestamps(tid, ts, insertedNodes, deletedNodes);\r\n \r\n if (!logicalDeletion) {\r\n // physical deletion will happen at the same time as logical deletion\r\n physical_deletion_succeeded(tid, deletedNodes);\r\n }\r\n \r\n#if defined USE_RQ_DEBUGGING\r\n DEBUG_RECORD_UPDATE_CHECKSUM(tid, ts, insertedNodes, deletedNodes, ds);\r\n#endif\r\n return lin_newval;\r\n }\r\n \r\n // replace the linearization point of an update that inserts or deletes nodes\r\n // with an invocation of this function if the linearization point is a CAS\r\n template \r\n inline T linearize_update_at_cas(\r\n const int tid,\r\n T volatile * const lin_addr,\r\n const T& lin_oldval,\r\n const T& lin_newval,\r\n NodeType * const * const insertedNodes,\r\n NodeType * const * const deletedNodes) {\r\n\r\n if (!logicalDeletion) {\r\n // physical deletion will happen at the same time as logical deletion\r\n announce_physical_deletion(tid, deletedNodes);\r\n }\r\n \r\n // htm path\r\n long long ts;\r\n T res;\r\n int limit = MAX_HTM_ATTEMPTS;\r\n while (limit--) {\r\n while (rwlock.isWriteLocked()) {}\r\n if (XBEGIN() == _XBEGIN_STARTED) {\r\n if (rwlock.isWriteLocked()) XABORT(1);\r\n ts = timestamp;\r\n res = __sync_val_compare_and_swap(lin_addr, lin_oldval, lin_newval); // original linearization point\r\n// res = *lin_addr; // manually implement CAS\r\n// if (res == lin_oldval) *lin_addr = lin_newval;\r\n XEND();\r\n ++threadData[tid].commitReader;\r\n goto committed;\r\n } else ++threadData[tid].abortReader;\r\n }\r\n \r\n // fallback path\r\n ++threadData[tid].fallback;\r\n rwlock.readLock();\r\n ts = timestamp;\r\n res = __sync_val_compare_and_swap(lin_addr, lin_oldval, lin_newval);\r\n rwlock.readUnlock();\r\n \r\ncommitted:\r\n if (res == lin_oldval){\r\n set_insertion_timestamps(tid, ts, insertedNodes, deletedNodes);\r\n set_deletion_timestamps(tid, ts, insertedNodes, deletedNodes);\r\n \r\n if (!logicalDeletion) {\r\n // physical deletion will happen at the same time as logical deletion\r\n physical_deletion_succeeded(tid, deletedNodes);\r\n }\r\n \r\n#if defined USE_RQ_DEBUGGING\r\n DEBUG_RECORD_UPDATE_CHECKSUM(tid, ts, insertedNodes, deletedNodes, ds);\r\n#endif\r\n } else {\r\n if (!logicalDeletion) {\r\n // physical deletion will happen at the same time as logical deletion\r\n physical_deletion_failed(tid, deletedNodes);\r\n } \r\n }\r\n return res;\r\n }\r\n\r\n // invoke at the start of each traversal\r\n inline void traversal_start(const int tid) {\r\n threadData[tid].hashlist->clear();\r\n\r\n// // htm path\r\n// int limit = MAX_HTM_ATTEMPTS;\r\n// while (limit--) {\r\n// while (rwlock.isLocked()) {}\r\n// if (XBEGIN() == _XBEGIN_STARTED) {\r\n// if (rwlock.isLocked()) XABORT(1);\r\n// threadData[tid].rq_lin_time = ++timestamp; // linearization point of range query (at the write to timestamp)\r\n// XEND();\r\n// ++threadData[tid].commitWriter;\r\n// goto committed;\r\n// } else ++threadData[tid].abortWriter;\r\n// }\r\n//committed:\r\n \r\n // fallback path\r\n rwlock.writeLock();\r\n threadData[tid].rq_lin_time = ++timestamp; // linearization point of range query (at the write to timestamp)\r\n rwlock.writeUnlock();\r\n }\r\n\r\nprivate:\r\n // invoke each time a traversal visits a node with a key in the desired range:\r\n // if the node belongs in the range query, it will be placed in rqResult[index]\r\n inline int __traversal_try_add(const int tid, NodeType * const node, NodeType ** const nodeSource, K * const outputKeys, V * const outputValues, const K& lo, const K& hi, bool foundDuringTraversal) {\r\n\r\n // rqResultKeys should have space for MAX_KEYS_PER_NODE keys, AT LEAST\r\n \r\n // in the following, rather than having deeply nested if-else blocks,\r\n // we return asap, and list facts that must be true if we didn't return\r\n assert(foundDuringTraversal || !logicalDeletion || ds->isLogicallyDeleted(tid, node));\r\n \r\n long long itime = TIMESTAMP_NOT_SET;\r\n while (itime == TIMESTAMP_NOT_SET) { itime = node->itime; }\r\n if (node->itime >= threadData[tid].rq_lin_time) return 0; // node was inserted after the range query\r\n // fact: node was inserted before the range query\r\n \r\n bool logicallyDeleted = (logicalDeletion && ds->isLogicallyDeleted(tid, node));\r\n long long dtime = TIMESTAMP_NOT_SET;\r\n\r\n if (!logicalDeletion && foundDuringTraversal) goto tryAddToRQ; // no logical deletion. since node was inserted before the range query, and the traversal encountered it, it must have been deleted AFTER the traversal encountered it.\r\n // fact: no logical deletion ==> did not find node during traversal\r\n\r\n dtime = node->dtime;\r\n if (dtime != TIMESTAMP_NOT_SET) {\r\n if (dtime < threadData[tid].rq_lin_time) return 0; // node was deleted before the range query\r\n goto tryAddToRQ;\r\n }\r\n\r\n // fact: dtime was not set above\r\n if (logicalDeletion && !logicallyDeleted) goto tryAddToRQ; // if logical deletion is used with marking, the fact that node was inserted before the range query, and that the traversal encountered node, is NOT enough to argue that node was in the data structure when the traversal started. why? when the traversal encountered node, it might have already been marked. so, we check if node is marked. if not, then the node has not yet been deleted.\r\n // fact: if there is logical deletion, then the node has now been deleted\r\n\r\n ///////////////////////// HANDLE UNKNOWN DTIME /////////////////////////\r\n // if we are executing this because node was ANNOUNCED by a process,\r\n // as something that MIGHT soon be deleted (if nodeSource != NULL),\r\n // then node might not ever actually be deleted,\r\n // so we can't spin forever on dtime.\r\n if (nodeSource != NULL) {\r\n while (dtime == TIMESTAMP_NOT_SET && *nodeSource == node) { dtime = node->dtime; }\r\n if (dtime == TIMESTAMP_NOT_SET) {\r\n // above loop exited because the process removed its announcement to this node!\r\n // if the process deleted the node, then it removed the\r\n // announcement AFTER setting dtime.\r\n // so we reread dtime one more time, to figure out whether\r\n // the process actually deleted the node.\r\n SOFTWARE_BARRIER; // prevent read of dtime from happening before last read of *nodeSource\r\n dtime = node->dtime;\r\n if (dtime == TIMESTAMP_NOT_SET) {\r\n // since dtime is not set, the process did NOT delete the node.\r\n // so, either a DIFFERENT process deleted it,\r\n // or it was found during the data structure traversal.\r\n // if another process deleted it, then we will find it\r\n // either in that process' announcements, or in a limbo bag.\r\n return 0;\r\n }\r\n // the node has been deleted, and dtime is set, so we check dtime below.\r\n }\r\n } else {\r\n while (dtime == TIMESTAMP_NOT_SET) { dtime = node->dtime; }\r\n }\r\n if (dtime < threadData[tid].rq_lin_time) return 0; // node was deleted before the range query\r\n // fact: node was inserted before the rq and deleted after it\r\n \r\n ///////////////////// TRY TO ADD NODE'S KEYS TO RQ /////////////////////\r\n // note: this way of organizing this decision tree favors trees with fat multi-key nodes, because getKeys is delayed as long as possible.\r\n \r\ntryAddToRQ:\r\n // fetch the node's keys that are in the set\r\n int cnt = ds->getKeys(tid, node, outputKeys, outputValues);\r\n assert(cnt < RQ_DEBUGGING_MAX_KEYS_PER_NODE);\r\n if (cnt == 0) return 0; // node doesn't contain any keys that are in the set\r\n // TODO: properly assert that getKeys doesn't run out of bounds on outputKeys[...] (i'm quite certain it doesn't, currently, though.)\r\n \r\n // note: in the following loop, we shift keys in the outputKeys array left to eliminate any that ultimately should not be added to the range query\r\n int numNewKeys = 0;\r\n for (int i=0;iisInRange(outputKeys[i], lo, hi)) goto doNotAddToRQ; // key is NOT in the desired range\r\n if (threadData[tid].hashlist->contains(outputKeys[i])) goto doNotAddToRQ; // key is already in the range query\r\n outputKeys[numNewKeys] = outputKeys[i]; // save this as a new key added to the RQ\r\n outputValues[numNewKeys] = outputValues[i];\r\n ++numNewKeys;\r\n\r\ndoNotAddToRQ: (0);\r\n }\r\n return numNewKeys;\r\n }\r\n \r\n inline void traversal_try_add(const int tid, NodeType * const node, NodeType ** const nodeSource, K * const rqResultKeys, V * const rqResultValues, int * const startIndex, const K& lo, const K& hi, bool foundDuringTraversal) {\r\n//#if defined MICROBENCH && !defined NDEBUG\r\n// assert(*startIndex < 2*RQSIZE); // note: this assert is a hack. it should be *startIndex < size of rqResultKeys\r\n// if (*startIndex >= RQSIZE) {\r\n// cout<<\"ERROR: *startIndex=\"<<(*startIndex)<<\" is unexpectedly greater than or equal to RQSIZE=\"<insert(rqResultKeys[(*startIndex)++]);\r\n }\r\n // note: the above increments startIndex\r\n#if defined MICROBENCH\r\n assert(*startIndex <= RQSIZE);\r\n#endif\r\n }\r\n \r\npublic:\r\n inline void traversal_try_add(const int tid, NodeType * const node, K * const rqResultKeys, V * const rqResultValues, int * const startIndex, const K& lo, const K& hi) {\r\n traversal_try_add(tid, node, NULL, rqResultKeys, rqResultValues, startIndex, lo, hi, true);\r\n }\r\n \r\n // invoke at the end of each traversal:\r\n // any nodes that were deleted during the traversal,\r\n // and were consequently missed during the traversal,\r\n // are placed in rqResult[index]\r\n void traversal_end(const int tid, K * const rqResultKeys, V * const rqResultValues, int * const startIndex, const K& lo, const K& hi) {\r\n // todo: possibly optimize by skipping entire blocks if there are many keys to skip (does not seem to be justifiable for 4 work threads and 4 range query threads)\r\n\r\n SOFTWARE_BARRIER;\r\n long long end_timestamp = timestamp;\r\n SOFTWARE_BARRIER;\r\n \r\n // collect nodes announced by other processes\r\n for (int otherTid=0;otherTid * all_bags[NUM_PROCESSES*NUMBER_OF_EPOCH_BAGS+1];\r\n vector> all_iterators;\r\n int numIterators = 0;\r\n for (int otherTid=0;otherTid * thread_bags[NUMBER_OF_EPOCH_BAGS+1];\r\n recmgr->get((NodeType *) NULL)->reclaim->getSafeBlockbags(otherTid, thread_bags);\r\n for (int i=0;thread_bags[i];++i) {\r\n all_bags[numIterators] = thread_bags[i];\r\n all_iterators.push_back(thread_bags[i]->begin());\r\n ++numIterators;\r\n }\r\n }\r\n \r\n int numSkippedInEpochBags = 0;\r\n int numVisitedInEpochBags = 0;\r\n for (int ix = 0; ix < numIterators; ++ix) {\r\n for (; all_iterators[ix] != all_bags[ix]->end(); all_iterators[ix]++) {\r\n NodeType * node = (*all_iterators[ix]);\r\n assert(node);\r\n\r\n ++numVisitedInEpochBags;\r\n ++numSkippedInEpochBags;\r\n\r\n long long dtime = node->dtime;\r\n if (dtime != TIMESTAMP_NOT_SET && dtime > end_timestamp) continue;\r\n\r\n --numSkippedInEpochBags;\r\n \r\n if (!(logicalDeletion && canRetireNodesLogicallyDeletedByOtherProcesses)) {\r\n // if we cannot retire nodes that are logically deleted\r\n // by other processes, then we always retire nodes in\r\n // order of increasing dtime values.\r\n // so, the blockbag will be ordered, which means that,\r\n // if dtime is before the RQ, then all remaining nodes\r\n // in this bag were deleted before the RQ.\r\n // so, in this case, we skip to the next bag.\r\n if (dtime != TIMESTAMP_NOT_SET && dtime < threadData[tid].rq_lin_time) break;\r\n }\r\n\r\n traversal_try_add(tid, node, NULL, rqResultKeys, rqResultValues, startIndex, lo, hi, false);\r\n }\r\n }\r\n\r\n#if defined MICROBENCH && !defined NDEBUG\r\n if (*startIndex > RQSIZE) {\r\n cout<<\"ERROR: *startIndex=\"<<(*startIndex)<<\" is unexpectedly greater than or equal to RQSIZE=\"<\r\n#include \r\n#include \r\n#include \r\n\r\n// the following define enables an optimization that i'm not sure is correct.\r\n//#define COLLECT_ANNOUNCEMENTS_FAST\r\n\r\ntemplate \r\nclass RQProvider {\r\nprivate:\r\n struct __rq_thread_data {\r\n #define __RQ_THREAD_DATA_SIZE 1024\r\n union {\r\n struct { // anonymous struct inside anonymous union means we don't need to type anything special to access these variables\r\n long long rq_lin_time;\r\n HashList * hashlist;\r\n volatile char padding0[PREFETCH_SIZE_BYTES];\r\n void * announcements[MAX_NODES_DELETED_ATOMICALLY+1];\r\n int numAnnouncements;\r\n };\r\n char bytes[__RQ_THREAD_DATA_SIZE]; // avoid false sharing\r\n };\r\n } __attribute__((aligned(__RQ_THREAD_DATA_SIZE)));\r\n\r\n #define TIMESTAMP_NOT_SET 0\r\n #define HASHLIST_INIT_CAPACITY_POW2 (1<<8)\r\n\r\n const int NUM_PROCESSES;\r\n volatile char padding0[PREFETCH_SIZE_BYTES];\r\n volatile long long timestamp = 1;\r\n volatile char padding1[PREFETCH_SIZE_BYTES];\r\n RWLock rwlock;\r\n volatile char padding2[PREFETCH_SIZE_BYTES];\r\n __rq_thread_data * threadData;\r\n \r\n DataStructure * ds;\r\n RecordManager * const recmgr;\r\n\r\n int init[MAX_TID_POW2] = {0,};\r\n\r\npublic:\r\n RQProvider(const int numProcesses, DataStructure * ds, RecordManager * recmgr) : NUM_PROCESSES(numProcesses), ds(ds), recmgr(recmgr) {\r\n threadData = new __rq_thread_data[numProcesses];\r\n DEBUG_INIT_RQPROVIDER(numProcesses);\r\n }\r\n\r\n ~RQProvider() {\r\n// for (int tid=0;tiddestroy();\r\n// delete threadData[tid].hashlist;\r\n// }\r\n delete[] threadData;\r\n DEBUG_DEINIT_RQPROVIDER(NUM_PROCESSES);\r\n }\r\n\r\n // invoke before a given thread can perform any rq_functions\r\n void initThread(const int tid) {\r\n if (init[tid]) return; else init[tid] = !init[tid];\r\n\r\n threadData[tid].hashlist = new HashList();\r\n threadData[tid].hashlist->init(HASHLIST_INIT_CAPACITY_POW2);\r\n threadData[tid].numAnnouncements = 0;\r\n for (int i=0;idestroy();\r\n delete threadData[tid].hashlist;\r\n DEBUG_DEINIT_THREAD(tid);\r\n }\r\n\r\n // invoke whenever a new node is created/initialized\r\n inline void init_node(const int tid, NodeType * const node) {\r\n node->itime = TIMESTAMP_NOT_SET;\r\n node->dtime = TIMESTAMP_NOT_SET;\r\n }\r\n\r\n // for each address addr that is modified by rq_linearize_update_at_write\r\n // or rq_linearize_update_at_cas, you must replace any initialization of addr\r\n // with invocations of rq_write_addr\r\n template \r\n inline void write_addr(const int tid, T volatile * const addr, const T val) {\r\n *addr = val;\r\n }\r\n\r\n // for each address addr that is modified by rq_linearize_update_at_write\r\n // or rq_linearize_update_at_cas, you must replace any reads of addr with\r\n // invocations of rq_read_addr\r\n template \r\n inline T read_addr(const int tid, T volatile * const addr) {\r\n return *addr;\r\n }\r\n\r\n // IF DATA STRUCTURE PERFORMS LOGICAL DELETION\r\n // run some time BEFORE the physical deletion of a node\r\n // whose key has ALREADY been logically deleted.\r\n void announce_physical_deletion(const int tid, NodeType * const * const deletedNodes) {\r\n int i;\r\n for (i=0;deletedNodes[i];++i) {\r\n threadData[tid].announcements[threadData[tid].numAnnouncements+i] = deletedNodes[i];\r\n }\r\n SOFTWARE_BARRIER;\r\n threadData[tid].numAnnouncements += i;\r\n assert(threadData[tid].numAnnouncements <= MAX_NODES_DELETED_ATOMICALLY);\r\n SOFTWARE_BARRIER;\r\n }\r\n\r\n // IF DATA STRUCTURE PERFORMS LOGICAL DELETION\r\n // run AFTER performing announce_physical_deletion,\r\n // if the cas that was trying to physically delete node failed.\r\n void physical_deletion_failed(const int tid, NodeType * const * const deletedNodes) {\r\n for (int i=0;deletedNodes[i];++i) {\r\n --threadData[tid].numAnnouncements;\r\n#ifdef COLLECT_ANNOUNCEMENTS_FAST\r\n threadData[tid].announcements[threadData[tid].numAnnouncements] = NULL;\r\n#endif\r\n }\r\n assert(threadData[tid].numAnnouncements >= 0);\r\n }\r\n \r\n // IF DATA STRUCTURE PERFORMS LOGICAL DELETION\r\n // run AFTER performing announce_physical_deletion,\r\n // if the cas that was trying to physically delete node succeeded.\r\n void physical_deletion_succeeded(const int tid, NodeType * const * const deletedNodes) {\r\n int i;\r\n for (i=0;deletedNodes[i];++i) {\r\n recmgr->retire(tid, deletedNodes[i]);\r\n }\r\n SOFTWARE_BARRIER; // ensure nodes are placed in the epoch bag BEFORE they are removed from announcements.\r\n threadData[tid].numAnnouncements -= i;\r\n assert(threadData[tid].numAnnouncements >= 0);\r\n }\r\n \r\nprivate:\r\n \r\n inline void set_insertion_timestamps(\r\n const int tid,\r\n const long long ts,\r\n NodeType * const * const insertedNodes,\r\n NodeType * const * const deletedNodes) {\r\n \r\n // set insertion timestamps\r\n // for each i_node in insertedNodes\r\n for (int i_nodeix=0;insertedNodes[i_nodeix];++i_nodeix) {\r\n insertedNodes[i_nodeix]->itime = ts;\r\n }\r\n }\r\n\r\n inline void set_deletion_timestamps(\r\n const int tid,\r\n const long long ts,\r\n NodeType * const * const insertedNodes,\r\n NodeType * const * const deletedNodes) {\r\n \r\n // set deletion timestamps\r\n // for each d_node in deletedNodes\r\n for (int d_nodeix=0;deletedNodes[d_nodeix];++d_nodeix) {\r\n deletedNodes[d_nodeix]->dtime = ts;\r\n }\r\n }\r\n \r\npublic:\r\n\r\n // replace the linearization point of an update that inserts or deletes nodes\r\n // with an invocation of this function if the linearization point is a WRITE\r\n template \r\n inline T linearize_update_at_write(\r\n const int tid,\r\n T volatile * const lin_addr,\r\n const T& lin_newval,\r\n NodeType * const * const insertedNodes,\r\n NodeType * const * const deletedNodes) {\r\n\r\n if (!logicalDeletion) {\r\n // physical deletion will happen at the same time as logical deletion\r\n announce_physical_deletion(tid, deletedNodes);\r\n }\r\n \r\n rwlock.readLock();\r\n long long ts = timestamp;\r\n *lin_addr = lin_newval; // original linearization point\r\n rwlock.readUnlock();\r\n\r\n set_insertion_timestamps(tid, ts, insertedNodes, deletedNodes);\r\n set_deletion_timestamps(tid, ts, insertedNodes, deletedNodes);\r\n \r\n if (!logicalDeletion) {\r\n // physical deletion will happen at the same time as logical deletion\r\n physical_deletion_succeeded(tid, deletedNodes);\r\n }\r\n \r\n#if defined USE_RQ_DEBUGGING\r\n DEBUG_RECORD_UPDATE_CHECKSUM(tid, ts, insertedNodes, deletedNodes, ds);\r\n#endif\r\n return lin_newval;\r\n }\r\n \r\n // replace the linearization point of an update that inserts or deletes nodes\r\n // with an invocation of this function if the linearization point is a CAS\r\n template \r\n inline T linearize_update_at_cas(\r\n const int tid,\r\n T volatile * const lin_addr,\r\n const T& lin_oldval,\r\n const T& lin_newval,\r\n NodeType * const * const insertedNodes,\r\n NodeType * const * const deletedNodes) {\r\n\r\n if (!logicalDeletion) {\r\n // physical deletion will happen at the same time as logical deletion\r\n announce_physical_deletion(tid, deletedNodes);\r\n }\r\n \r\n rwlock.readLock();\r\n long long ts = timestamp;\r\n T res = __sync_val_compare_and_swap(lin_addr, lin_oldval, lin_newval);\r\n rwlock.readUnlock();\r\n \r\n if (res == lin_oldval){\r\n set_insertion_timestamps(tid, ts, insertedNodes, deletedNodes);\r\n set_deletion_timestamps(tid, ts, insertedNodes, deletedNodes);\r\n \r\n if (!logicalDeletion) {\r\n // physical deletion will happen at the same time as logical deletion\r\n physical_deletion_succeeded(tid, deletedNodes);\r\n }\r\n \r\n#if defined USE_RQ_DEBUGGING\r\n DEBUG_RECORD_UPDATE_CHECKSUM(tid, ts, insertedNodes, deletedNodes, ds);\r\n#endif\r\n } else {\r\n if (!logicalDeletion) {\r\n // physical deletion will happen at the same time as logical deletion\r\n physical_deletion_failed(tid, deletedNodes);\r\n } \r\n }\r\n return res;\r\n }\r\n\r\n // invoke at the start of each traversal\r\n inline void traversal_start(const int tid) {\r\n threadData[tid].hashlist->clear();\r\n rwlock.writeLock();\r\n threadData[tid].rq_lin_time = ++timestamp; // linearization point of range query (at the write to timestamp)\r\n rwlock.writeUnlock();\r\n }\r\n\r\nprivate:\r\n // invoke each time a traversal visits a node with a key in the desired range:\r\n // if the node belongs in the range query, it will be placed in rqResult[index]\r\n inline int __traversal_try_add(const int tid, NodeType * const node, NodeType ** const nodeSource, K * const outputKeys, V * const outputValues, const K& lo, const K& hi, bool foundDuringTraversal) {\r\n \r\n // rqResultKeys should have space for MAX_KEYS_PER_NODE keys, AT LEAST\r\n \r\n // in the following, rather than having deeply nested if-else blocks,\r\n // we return asap, and list facts that must be true if we didn't return\r\n assert(foundDuringTraversal || !logicalDeletion || ds->isLogicallyDeleted(tid, node));\r\n \r\n long long itime = TIMESTAMP_NOT_SET;\r\n while (itime == TIMESTAMP_NOT_SET) { itime = node->itime; }\r\n if (node->itime >= threadData[tid].rq_lin_time) return 0; // node was inserted after the range query\r\n // fact: node was inserted before the range query\r\n \r\n bool logicallyDeleted = (logicalDeletion && ds->isLogicallyDeleted(tid, node));\r\n long long dtime = TIMESTAMP_NOT_SET;\r\n\r\n if (!logicalDeletion && foundDuringTraversal) goto tryAddToRQ; // no logical deletion. since node was inserted before the range query, and the traversal encountered it, it must have been deleted AFTER the traversal encountered it.\r\n // fact: no logical deletion ==> did not find node during traversal\r\n\r\n dtime = node->dtime;\r\n if (dtime != TIMESTAMP_NOT_SET) {\r\n if (dtime < threadData[tid].rq_lin_time) return 0; // node was deleted before the range query\r\n goto tryAddToRQ;\r\n }\r\n\r\n // fact: dtime was not set above\r\n if (logicalDeletion && !logicallyDeleted) goto tryAddToRQ; // if logical deletion is used with marking, the fact that node was inserted before the range query, and that the traversal encountered node, is NOT enough to argue that node was in the data structure when the traversal started. why? when the traversal encountered node, it might have already been marked. so, we check if node is marked. if not, then the node has not yet been deleted.\r\n // fact: if there is logical deletion, then the node has now been deleted\r\n\r\n ///////////////////////// HANDLE UNKNOWN DTIME /////////////////////////\r\n // if we are executing this because node was ANNOUNCED by a process,\r\n // as something that MIGHT soon be deleted (if nodeSource != NULL),\r\n // then node might not ever actually be deleted,\r\n // so we can't spin forever on dtime.\r\n if (nodeSource != NULL) {\r\n while (dtime == TIMESTAMP_NOT_SET && *nodeSource == node) { dtime = node->dtime; }\r\n if (dtime == TIMESTAMP_NOT_SET) {\r\n // above loop exited because the process removed its announcement to this node!\r\n // if the process deleted the node, then it removed the\r\n // announcement AFTER setting dtime.\r\n // so we reread dtime one more time, to figure out whether\r\n // the process actually deleted the node.\r\n SOFTWARE_BARRIER; // prevent read of dtime from happening before last read of *nodeSource\r\n dtime = node->dtime;\r\n if (dtime == TIMESTAMP_NOT_SET) {\r\n // since dtime is not set, the process did NOT delete the node.\r\n // so, either a DIFFERENT process deleted it,\r\n // or it was found during the data structure traversal.\r\n // if another process deleted it, then we will find it\r\n // either in that process' announcements, or in a limbo bag.\r\n return 0;\r\n }\r\n // the node has been deleted, and dtime is set, so we check dtime below.\r\n }\r\n } else {\r\n while (dtime == TIMESTAMP_NOT_SET) { dtime = node->dtime; }\r\n }\r\n if (dtime < threadData[tid].rq_lin_time) return 0; // node was deleted before the range query\r\n // fact: node was inserted before the rq and deleted after it\r\n \r\n ///////////////////// TRY TO ADD NODE'S KEYS TO RQ /////////////////////\r\n // note: this way of organizing this decision tree favors trees with fat multi-key nodes, because getKeys is delayed as long as possible.\r\n \r\ntryAddToRQ:\r\n // fetch the node's keys that are in the set\r\n int cnt = ds->getKeys(tid, node, outputKeys, outputValues);\r\n assert(cnt < RQ_DEBUGGING_MAX_KEYS_PER_NODE);\r\n if (cnt == 0) return 0; // node doesn't contain any keys that are in the set\r\n // TODO: properly assert that getKeys doesn't run out of bounds on outputKeys[...] (i'm quite certain it doesn't, currently, though.)\r\n \r\n // note: in the following loop, we shift keys in the outputKeys array left to eliminate any that ultimately should not be added to the range query\r\n int numNewKeys = 0;\r\n for (int i=0;iisInRange(outputKeys[i], lo, hi)) goto doNotAddToRQ; // key is NOT in the desired range\r\n if (threadData[tid].hashlist->contains(outputKeys[i])) goto doNotAddToRQ; // key is already in the range query\r\n outputKeys[numNewKeys] = outputKeys[i]; // save this as a new key added to the RQ\r\n outputValues[numNewKeys] = outputValues[i];\r\n ++numNewKeys;\r\n\r\ndoNotAddToRQ: (0);\r\n }\r\n return numNewKeys;\r\n }\r\n \r\n inline void traversal_try_add(const int tid, NodeType * const node, NodeType ** const nodeSource, K * const rqResultKeys, V * const rqResultValues, int * const startIndex, const K& lo, const K& hi, bool foundDuringTraversal) {\r\n//#if defined MICROBENCH && !defined NDEBUG\r\n// assert(*startIndex < 2*RQSIZE); // note: this assert is a hack. it should be *startIndex < size of rqResultKeys\r\n// if (*startIndex >= RQSIZE) {\r\n// cout<<\"ERROR: *startIndex=\"<<(*startIndex)<<\" is unexpectedly greater than or equal to RQSIZE=\"<insert(rqResultKeys[(*startIndex)++]);\r\n }\r\n // note: the above increments startIndex\r\n#if defined MICROBENCH\r\n assert(*startIndex <= RQSIZE);\r\n#endif\r\n }\r\n\r\npublic:\r\n inline void traversal_try_add(const int tid, NodeType * const node, K * const rqResultKeys, V * const rqResultValues, int * const startIndex, const K& lo, const K& hi) {\r\n traversal_try_add(tid, node, NULL, rqResultKeys, rqResultValues, startIndex, lo, hi, true);\r\n }\r\n \r\n // invoke at the end of each traversal:\r\n // any nodes that were deleted during the traversal,\r\n // and were consequently missed during the traversal,\r\n // are placed in rqResult[index]\r\n void traversal_end(const int tid, K * const rqResultKeys, V * const rqResultValues, int * const startIndex, const K& lo, const K& hi) {\r\n // todo: possibly optimize by skipping entire blocks if there are many keys to skip (does not seem to be justifiable for 4 work threads and 4 range query threads)\r\n\r\n SOFTWARE_BARRIER;\r\n long long end_timestamp = timestamp;\r\n SOFTWARE_BARRIER;\r\n \r\n // collect nodes announced by other processes\r\n#ifdef COLLECT_ANNOUNCEMENTS_FAST\r\n int numCollected = 0;\r\n NodeType * collectedAnnouncement[NUM_PROCESSES*MAX_NODES_DELETED_ATOMICALLY];\r\n NodeType ** announcementSource[NUM_PROCESSES*MAX_NODES_DELETED_ATOMICALLY];\r\n#endif\r\n for (int otherTid=0;otherTid * all_bags[NUM_PROCESSES*NUMBER_OF_EPOCH_BAGS+1];\r\n vector> all_iterators;\r\n int numIterators = 0;\r\n for (int otherTid=0;otherTid * thread_bags[NUMBER_OF_EPOCH_BAGS+1];\r\n recmgr->get((NodeType *) NULL)->reclaim->getSafeBlockbags(otherTid, thread_bags);\r\n for (int i=0;thread_bags[i];++i) {\r\n all_bags[numIterators] = thread_bags[i];\r\n all_iterators.push_back(thread_bags[i]->begin());\r\n ++numIterators;\r\n }\r\n }\r\n \r\n#ifdef COLLECT_ANNOUNCEMENTS_FAST\r\n // try to add nodes collected from process announcements to the RQ\r\n for (int i=0;iend(); all_iterators[ix]++) {\r\n NodeType * node = (*all_iterators[ix]);\r\n assert(node);\r\n\r\n ++numVisitedInEpochBags;\r\n ++numSkippedInEpochBags;\r\n\r\n long long dtime = node->dtime;\r\n if (dtime != TIMESTAMP_NOT_SET && dtime > end_timestamp) continue;\r\n\r\n --numSkippedInEpochBags;\r\n\r\n if (!(logicalDeletion && canRetireNodesLogicallyDeletedByOtherProcesses)) {\r\n // if we cannot retire nodes that are logically deleted\r\n // by other processes, then we always retire nodes in\r\n // order of increasing dtime values.\r\n // so, the blockbag will be ordered, which means that,\r\n // if dtime is before the RQ, then all remaining nodes\r\n // in this bag were deleted before the RQ.\r\n // so, in this case, we skip to the next bag.\r\n if (dtime != TIMESTAMP_NOT_SET && dtime < threadData[tid].rq_lin_time) break;\r\n }\r\n\r\n traversal_try_add(tid, node, NULL, rqResultKeys, rqResultValues, startIndex, lo, hi, false);\r\n }\r\n }\r\n\r\n#if defined MICROBENCH && !defined NDEBUG\r\n if (*startIndex > RQSIZE) {\r\n cout<<\"ERROR: *startIndex=\"<<(*startIndex)<<\" is unexpectedly greater than or equal to RQSIZE=\"<\r\n#include \r\n#include \r\n#include \"snapcollector.h\"\r\n\r\ntemplate \r\nclass RQProvider {\r\nprivate:\r\n struct __rq_thread_data {\r\n #define __RQ_THREAD_DATA_SIZE 1024\r\n union {\r\n struct { // anonymous struct inside anonymous union means we don't need to type anything special to access these variables\r\n long long rq_lin_time;\r\n SnapCollector * currentSnapCollector;\r\n SnapCollector * snapCollectorToRetire;\r\n };\r\n char bytes[__RQ_THREAD_DATA_SIZE]; // avoid false sharing\r\n };\r\n } __attribute__((aligned(__RQ_THREAD_DATA_SIZE)));\r\n\r\n const int NUM_PROCESSES;\r\n volatile long long timestamp = 1;\r\n pthread_rwlock_t rwlock;\r\n __rq_thread_data * threadData;\r\n \r\n DataStructure * const ds;\r\n RecordManager * const recmgr;\r\n volatile char padding[PREFETCH_SIZE_BYTES];\r\n SnapCollector * volatile snapPointer;\r\n \r\n int init[MAX_TID_POW2] = {0,};\r\n\r\npublic:\r\n RQProvider(const int numProcesses, DataStructure * ds, RecordManager * recmgr) : NUM_PROCESSES(numProcesses), ds(ds), recmgr(recmgr) {\r\n assert(logicalDeletion); // Timnat's iterator algorithm REQUIRES logical deletion!\r\n if (pthread_rwlock_init(&rwlock, NULL)) error(\"could not init rwlock\");\r\n threadData = new __rq_thread_data[numProcesses];\r\n \r\n const int dummyTid = 0;\r\n \r\n recmgr->initThread(dummyTid); // must initialize record manager before allocating!!\r\n initThread(dummyTid);\r\n \r\n // initialize dummy snap collector\r\n snapPointer = recmgr->template allocate >(dummyTid);\r\n#ifdef __HANDLE_STATS\r\n GSTATS_APPEND(dummyTid, extra_type1_allocated_addresses, ((long long) snapPointer)%(1<<12));\r\n#endif\r\n snapPointer->init(dummyTid, numProcesses, recmgr, ds->KEY_MIN, ds->KEY_MAX+1);\r\n snapPointer->BlockFurtherPointers(dummyTid, recmgr);\r\n snapPointer->Deactivate(NULL, NULL, NULL);\r\n snapPointer->BlockFurtherReports();\r\n \r\n DEBUG_INIT_RQPROVIDER(numProcesses);\r\n }\r\n\r\n ~RQProvider() {\r\n if (pthread_rwlock_destroy(&rwlock)) error(\"could not destroy rwlock\");\r\n delete[] threadData;\r\n snapPointer->retire(0 /* dummy tid */, recmgr);\r\n DEBUG_DEINIT_RQPROVIDER(NUM_PROCESSES);\r\n }\r\n\r\n // invoke before a given thread can perform any rq_functions\r\n void initThread(const int tid) {\r\n if (init[tid]) return; else init[tid] = !init[tid];\r\n\r\n threadData[tid].rq_lin_time = 0;\r\n threadData[tid].currentSnapCollector = NULL;\r\n threadData[tid].snapCollectorToRetire = NULL;\r\n DEBUG_INIT_THREAD(tid);\r\n }\r\n\r\n // invoke once a given thread will no longer perform any rq_ functions\r\n void deinitThread(const int tid) {\r\n if (!init[tid]) return; else init[tid] = !init[tid];\r\n\r\n DEBUG_DEINIT_THREAD(tid);\r\n }\r\n\r\n // invoke whenever a new node is created/initialized\r\n inline void init_node(const int tid, NodeType * const node) {}\r\n\r\n // for each address addr that is modified by rq_linearize_update_at_write\r\n // or rq_linearize_update_at_cas, you must replace any initialization of addr\r\n // with invocations of rq_write_addr\r\n template \r\n inline void write_addr(const int tid, T volatile * const addr, const T val) {\r\n *addr = val;\r\n }\r\n\r\n // for each address addr that is modified by rq_linearize_update_at_write\r\n // or rq_linearize_update_at_cas, you must replace any reads of addr with\r\n // invocations of rq_read_addr\r\n template \r\n inline T read_addr(const int tid, T volatile * const addr) {\r\n return *addr;\r\n }\r\n\r\n /**\r\n * Added function only for Timnat's SnapCollector.\r\n * This must be invoked just before the return statement of every search.\r\n */\r\n inline void search_report_target_key(const int tid, const K key, NodeType * const node) {\r\n SnapCollector * sc = snapPointer;\r\n if (sc->IsActive()) {\r\n ReportType type = ds->isLogicallyDeleted(tid, node) ? ReportType::Remove : ReportType::Add;\r\n sc->Report(tid, node, type, key, recmgr);\r\n }\r\n SOFTWARE_BARRIER;\r\n }\r\n \r\n /**\r\n * Added function only for Timnat's SnapCollector.\r\n * This must be invoked just before the return statement of every insertion\r\n * that does not modify the data structure.\r\n */\r\n inline void insert_readonly_report_target_key(const int tid, NodeType * const node) {\r\n SnapCollector * sc = snapPointer;\r\n if (sc->IsActive()) {\r\n if (!ds->isLogicallyDeleted(tid, node)) {\r\n sc->Report(tid, node, ReportType::Add, node->key, recmgr);\r\n }\r\n }\r\n SOFTWARE_BARRIER;\r\n }\r\n \r\n /**\r\n * Added function only for Timnat's SnapCollector.\r\n * This can be invoked to determine if the current SnapCollector is active.\r\n */\r\n inline bool traversal_is_active(const int tid) {\r\n return threadData[tid].currentSnapCollector->IsActive();\r\n }\r\n \r\nprivate:\r\n inline void delete_report_target_key(const int tid, NodeType * const node) {\r\n if (node) {\r\n SnapCollector * sc = snapPointer;\r\n if (sc->IsActive()) {\r\n sc->Report(tid, node, ReportType::Remove, node->key, recmgr);\r\n }\r\n SOFTWARE_BARRIER;\r\n }\r\n }\r\n \r\npublic:\r\n \r\n // IF DATA STRUCTURE PERFORMS LOGICAL DELETION\r\n // run some time BEFORE the physical deletion of a node\r\n // whose key has ALREADY been logically deleted.\r\n inline void announce_physical_deletion(const int tid, NodeType * const * const deletedNodes) {\r\n assert(!deletedNodes[0] || !deletedNodes[1]);\r\n delete_report_target_key(tid, deletedNodes[0]);\r\n }\r\n\r\n // IF DATA STRUCTURE PERFORMS LOGICAL DELETION\r\n // run AFTER performing announce_physical_deletion,\r\n // if the cas that was trying to physically delete node failed.\r\n inline void physical_deletion_failed(const int tid, NodeType * const * const deletedNodes) {}\r\n \r\n // IF DATA STRUCTURE PERFORMS LOGICAL DELETION\r\n // run AFTER performing announce_physical_deletion,\r\n // if the cas that was trying to physically delete node succeeded.\r\n inline void physical_deletion_succeeded(const int tid, NodeType * const * const deletedNodes) {\r\n int i;\r\n for (i=0;deletedNodes[i];++i) {\r\n recmgr->retire(tid, deletedNodes[i]);\r\n }\r\n }\r\n \r\n // replace the linearization point of an update that inserts or deletes nodes\r\n // with an invocation of this function if the linearization point is a WRITE\r\n template \r\n inline T linearize_update_at_write(\r\n const int tid,\r\n T volatile * const lin_addr,\r\n const T& lin_newval,\r\n NodeType * const * const insertedNodes,\r\n NodeType * const * const deletedNodes) {\r\n \r\n assert((insertedNodes[0] && !deletedNodes[0])\r\n || (!insertedNodes[0] && deletedNodes[0]));\r\n\r\n#ifdef RQ_USE_TIMESTAMPS\r\n if (pthread_rwlock_rdlock(&rwlock)) error(\"could not read-lock rwlock\");\r\n long long ts = timestamp;\r\n#else\r\n long long ts = 1;\r\n#endif\r\n\r\n *lin_addr = lin_newval; // original linearization point\r\n#ifdef RQ_USE_TIMESTAMPS\r\n if (pthread_rwlock_unlock(&rwlock)) error(\"could not read-unlock rwlock\");\r\n#endif\r\n \r\n if (insertedNodes[0]) insert_readonly_report_target_key(tid, insertedNodes[0]);\r\n if (deletedNodes[0]) delete_report_target_key(tid, deletedNodes[0]);\r\n\r\n#if defined USE_RQ_DEBUGGING\r\n DEBUG_RECORD_UPDATE_CHECKSUM(tid, ts, insertedNodes, deletedNodes, ds);\r\n#endif\r\n return lin_newval;\r\n }\r\n \r\n // replace the linearization point of an update that inserts or deletes nodes\r\n // with an invocation of this function if the linearization point is a CAS\r\n template \r\n inline T linearize_update_at_cas(\r\n const int tid,\r\n T volatile * const lin_addr,\r\n const T& lin_oldval,\r\n const T& lin_newval,\r\n NodeType * const * const insertedNodes,\r\n NodeType * const * const deletedNodes) {\r\n\r\n assert((insertedNodes[0] && !deletedNodes[0])\r\n || (!insertedNodes[0] && deletedNodes[0]));\r\n\r\n#ifdef RQ_USE_TIMESTAMPS\r\n if (pthread_rwlock_rdlock(&rwlock)) error(\"could not read-lock rwlock\");\r\n long long ts = timestamp;\r\n#else\r\n long long ts = 1;\r\n#endif\r\n \r\n T res = __sync_val_compare_and_swap(lin_addr, lin_oldval, lin_newval);\r\n#ifdef RQ_USE_TIMESTAMPS\r\n if (pthread_rwlock_unlock(&rwlock)) error(\"could not read-unlock rwlock\");\r\n#endif\r\n \r\n if (res == lin_oldval){\r\n if (insertedNodes[0]) insert_readonly_report_target_key(tid, insertedNodes[0]);\r\n if (deletedNodes[0]) delete_report_target_key(tid, deletedNodes[0]);\r\n\r\n#if defined USE_RQ_DEBUGGING\r\n DEBUG_RECORD_UPDATE_CHECKSUM(tid, ts, insertedNodes, deletedNodes, ds);\r\n#endif\r\n }\r\n return res;\r\n }\r\n\r\n // invoke at the start of each traversal\r\n inline void traversal_start(const int tid) {\r\n#if !defined(RQ_USE_TIMESTAMPS)\r\n threadData[tid].rq_lin_time = 1;\r\n#endif \r\n\r\n threadData[tid].currentSnapCollector = snapPointer;\r\n SOFTWARE_BARRIER;\r\n if (!threadData[tid].currentSnapCollector->IsActive()) {\r\n SnapCollector * candidate = recmgr->template allocate >(tid);\r\n#ifdef __HANDLE_STATS\r\n GSTATS_APPEND(tid, extra_type1_allocated_addresses, ((long long) candidate)%(1<<12));\r\n#endif\r\n candidate->init(tid, NUM_PROCESSES, recmgr, ds->KEY_MIN, ds->KEY_MAX+1);\r\n if (__sync_bool_compare_and_swap(&snapPointer, threadData[tid].currentSnapCollector, candidate)) {\r\n // delay retiring until later, because we've started accepting reports,\r\n // and we don't want to waste time while we are accepting reports,\r\n // because we don't want to receive many reports...\r\n threadData[tid].snapCollectorToRetire = threadData[tid].currentSnapCollector;\r\n threadData[tid].currentSnapCollector = candidate;\r\n } else {\r\n candidate->retire(tid, recmgr);\r\n threadData[tid].currentSnapCollector = snapPointer;\r\n }\r\n }\r\n// usleep(200000);\r\n }\r\n\r\n inline NodeType * traversal_try_add(const int tid, NodeType * const node, K * const rqResultKeys, V * const rqResultValues, int * const startIndex, const K& lo, const K& hi) {\r\n SnapCollector * sc = threadData[tid].currentSnapCollector;\r\n return sc->AddNode(tid, node, node->key, recmgr);\r\n }\r\n \r\n // invoke at the end of each traversal:\r\n // any nodes that were deleted during the traversal,\r\n // and were consequently missed during the traversal,\r\n // are placed in rqResult[index]\r\n void traversal_end(const int tid, K * const rqResultKeys, V * const rqResultValues, int * const startIndex, const K& lo, const K& hi) {\r\n SnapCollector * sc = threadData[tid].currentSnapCollector;\r\n\r\n sc->BlockFurtherPointers(tid, recmgr);\r\n SOFTWARE_BARRIER;\r\n sc->Deactivate(NULL, NULL, NULL);\r\n sc->BlockFurtherReports();\r\n SOFTWARE_BARRIER;\r\n\r\n sc->Prepare(tid, recmgr);\r\n \r\n NodeType * curr = NULL;\r\n while ((curr = sc->GetNext(tid))) {\r\n if (curr->key < lo) continue;\r\n if (curr->key > hi) break;\r\n rqResultKeys[*startIndex] = curr->key;\r\n rqResultValues[*startIndex] = curr->val;\r\n ++*startIndex;\r\n }\r\n#if defined MICROBENCH\r\n assert(*startIndex <= RQSIZE);\r\n#endif\r\n \r\n#ifdef SNAPCOLLECTOR_PRINT_RQS\r\n// for (int i=0;i<*startIndex;++i) {\r\n// cout<<\" \"<retire(tid, recmgr);\r\n threadData[tid].snapCollectorToRetire = NULL;\r\n }\r\n }\r\n};\r\n\r\n#endif\t/* RQ_RWLOCK_H */\r\n\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/rq/rq_unsafe.h", "content": "/* \r\n * File: rq_unsafe.h\r\n * Author: trbot\r\n *\r\n * Created on May 15, 2017, 5:06 PM\r\n */\r\n\r\n#ifndef RQ_UNSAFE_H\r\n#define\tRQ_UNSAFE_H\r\n\r\n#include \"rq_debugging.h\"\r\n#include \r\n#include \r\n\r\n#ifndef casword_t\r\n#define casword_t uintptr_t\r\n#endif\r\n\r\ntemplate \r\nclass RQProvider {\r\nprivate:\r\n struct __rq_thread_data {\r\n #define __RQ_THREAD_DATA_SIZE 1024\r\n union {\r\n struct { // anonymous struct inside anonymous union means we don't need to type anything special to access these variables\r\n long long rq_lin_time;\r\n };\r\n char bytes[__RQ_THREAD_DATA_SIZE]; // avoid false sharing\r\n };\r\n } __attribute__((aligned(__RQ_THREAD_DATA_SIZE)));\r\n\r\n #define TIMESTAMP_NOT_SET 0\r\n \r\n const int NUM_PROCESSES;\r\n volatile char padding0[PREFETCH_SIZE_BYTES];\r\n volatile long long timestamp = 1;\r\n volatile char padding1[PREFETCH_SIZE_BYTES];\r\n RWLock rwlock;\r\n volatile char padding2[PREFETCH_SIZE_BYTES];\r\n __rq_thread_data * threadData;\r\n \r\n DataStructure * ds;\r\n RecordManager * const recmgr;\r\n\r\n int init[MAX_TID_POW2] = {0,};\r\n\r\npublic:\r\n RQProvider(const int numProcesses, DataStructure * ds, RecordManager * recmgr) : NUM_PROCESSES(numProcesses), ds(ds), recmgr(recmgr) {\r\n threadData = new __rq_thread_data[numProcesses];\r\n DEBUG_INIT_RQPROVIDER(numProcesses);\r\n }\r\n\r\n ~RQProvider() {\r\n delete[] threadData;\r\n DEBUG_DEINIT_RQPROVIDER(NUM_PROCESSES);\r\n }\r\n\r\n // invoke before a given thread can perform any rq_functions\r\n void initThread(const int tid) {\r\n if (init[tid]) return; else init[tid] = !init[tid];\r\n\r\n DEBUG_INIT_THREAD(tid);\r\n }\r\n\r\n // invoke once a given thread will no longer perform any rq_ functions\r\n void deinitThread(const int tid) {\r\n if (!init[tid]) return; else init[tid] = !init[tid];\r\n\r\n DEBUG_DEINIT_THREAD(tid);\r\n }\r\n\r\n // invoke whenever a new node is created/initialized\r\n inline void init_node(const int tid, NodeType * const node) {}\r\n\r\n // for each address addr that is modified by rq_linearize_update_at_write\r\n // or rq_linearize_update_at_cas, you must replace any initialization of addr\r\n // with invocations of rq_write_addr\r\n template \r\n inline void write_addr(const int tid, T volatile * const addr, const T val) {\r\n *addr = val;\r\n }\r\n\r\n // for each address addr that is modified by rq_linearize_update_at_write\r\n // or rq_linearize_update_at_cas, you must replace any reads of addr with\r\n // invocations of rq_read_addr\r\n template \r\n inline T read_addr(const int tid, T volatile * const addr) {\r\n return *addr;\r\n }\r\n \r\n // IF DATA STRUCTURE PERFORMS LOGICAL DELETION\r\n // run some time BEFORE the physical deletion of a node\r\n // whose key has ALREADY been logically deleted.\r\n inline void announce_physical_deletion(const int tid, NodeType * const * const deletedNodes) {}\r\n\r\n // IF DATA STRUCTURE PERFORMS LOGICAL DELETION\r\n // run AFTER performing announce_physical_deletion,\r\n // if the cas that was trying to physically delete node failed.\r\n inline void physical_deletion_failed(const int tid, NodeType * const * const deletedNodes) {}\r\n \r\n // IF DATA STRUCTURE PERFORMS LOGICAL DELETION\r\n // run AFTER performing announce_physical_deletion,\r\n // if the cas that was trying to physically delete node succeeded.\r\n inline void physical_deletion_succeeded(const int tid, NodeType * const * const deletedNodes) {\r\n int i;\r\n for (i=0;deletedNodes[i];++i) {\r\n recmgr->retire(tid, deletedNodes[i]);\r\n }\r\n }\r\n\r\n // replace the linearization point of an update that inserts or deletes nodes\r\n // with an invocation of this function if the linearization point is a WRITE\r\n template \r\n inline T linearize_update_at_write(\r\n const int tid,\r\n T volatile * const lin_addr,\r\n const T& lin_newval,\r\n NodeType * const * const insertedNodes,\r\n NodeType * const * const deletedNodes) {\r\n\r\n if (!logicalDeletion) {\r\n // physical deletion will happen at the same time as logical deletion\r\n announce_physical_deletion(tid, deletedNodes);\r\n }\r\n \r\n#ifdef RQ_USE_TIMESTAMPS\r\n rwlock.readLock();\r\n long long ts = timestamp;\r\n#else\r\n long long ts = 1;\r\n#endif\r\n\r\n *lin_addr = lin_newval; // original linearization point\r\n#ifdef RQ_USE_TIMESTAMPS\r\n rwlock.readUnlock();\r\n#endif\r\n\r\n if (!logicalDeletion) {\r\n // physical deletion will happen at the same time as logical deletion\r\n physical_deletion_succeeded(tid, deletedNodes);\r\n }\r\n \r\n#if defined USE_RQ_DEBUGGING\r\n DEBUG_RECORD_UPDATE_CHECKSUM(tid, ts, insertedNodes, deletedNodes, ds);\r\n#endif\r\n return lin_newval;\r\n }\r\n\r\n // replace the linearization point of an update that inserts or deletes nodes\r\n // with an invocation of this function if the linearization point is a CAS\r\n template \r\n inline T linearize_update_at_cas(\r\n const int tid,\r\n T volatile * const lin_addr,\r\n const T& lin_oldval,\r\n const T& lin_newval,\r\n NodeType * const * const insertedNodes,\r\n NodeType * const * const deletedNodes) {\r\n\r\n if (!logicalDeletion) {\r\n // physical deletion will happen at the same time as logical deletion\r\n announce_physical_deletion(tid, deletedNodes);\r\n }\r\n \r\n#ifdef RQ_USE_TIMESTAMPS\r\n rwlock.readLock();\r\n long long ts = timestamp;\r\n#else\r\n long long ts = 1;\r\n#endif\r\n T res = __sync_val_compare_and_swap(lin_addr, lin_oldval, lin_newval); // original linearization point\r\n#ifdef RQ_USE_TIMESTAMPS\r\n rwlock.readUnlock();\r\n#endif\r\n\r\n if (res == lin_oldval) {\r\n if (!logicalDeletion) {\r\n // physical deletion will happen at the same time as logical deletion\r\n physical_deletion_succeeded(tid, deletedNodes);\r\n }\r\n\r\n#if defined USE_RQ_DEBUGGING\r\n DEBUG_RECORD_UPDATE_CHECKSUM(tid, ts, insertedNodes, deletedNodes, ds);\r\n#endif\r\n } else {\r\n if (!logicalDeletion) {\r\n // physical deletion will happen at the same time as logical deletion\r\n physical_deletion_failed(tid, deletedNodes);\r\n }\r\n }\r\n \r\n return res;\r\n }\r\n\r\n // invoke at the start of each traversal\r\n inline void traversal_start(const int tid) {\r\n#ifdef RQ_USE_TIMESTAMPS\r\n rwlock.writeLock();\r\n threadData[tid].rq_lin_time = ++timestamp; // linearization point of range query (at the write to timestamp)\r\n rwlock.writeUnlock();\r\n#endif\r\n }\r\n\r\n // invoke each time a traversal visits a node with a key in the desired range:\r\n // if the node belongs in the range query, it will be placed in rqResult[index]\r\n inline void traversal_try_add(const int tid, NodeType * const node, K * const rqResultKeys, V * const rqResultValues, int * const startIndex, const K& lo, const K& hi) {\r\n int start = (*startIndex);\r\n int keysInNode = ds->getKeys(tid, node, rqResultKeys+start, rqResultValues+start);\r\n assert(keysInNode < RQ_DEBUGGING_MAX_KEYS_PER_NODE);\r\n if (keysInNode == 0) return;\r\n int location = start; \r\n for (int i=start;iisInRange(rqResultKeys[i], lo, hi)){\r\n rqResultKeys[location] = rqResultKeys[i];\r\n rqResultValues[location] = rqResultValues[i];\r\n ++location;\r\n } \r\n }\r\n *startIndex = location;\r\n#if defined MICROBENCH\r\n assert(*startIndex <= RQSIZE);\r\n#endif\r\n }\r\n\r\n // invoke at the end of each traversal:\r\n // any nodes that were deleted during the traversal,\r\n // and were consequently missed during the traversal,\r\n // are placed in rqResult[index]\r\n inline void traversal_end(const int tid, K * const rqResultKeys, V * const rqResultValues, int * const startIndex, const K& lo, const K& hi) {\r\n DEBUG_RECORD_RQ_SIZE(*startIndex);\r\n DEBUG_RECORD_RQ_CHECKSUM(tid, threadData[tid].rq_lin_time, rqResultKeys, *startIndex);\r\n }\r\n};\r\n\r\n#endif\t/* RQ_UNSAFE_H */\r\n\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/rq/snapcollector/reportitem.h", "content": "/* \r\n * File: reportitem.h\r\n * Author: trbot\r\n *\r\n * Created on June 21, 2017, 4:47 PM\r\n */\r\n\r\n#ifndef REPORTITEM_H\r\n#define REPORTITEM_H\r\n\r\nenum ReportType {Add, Remove};\r\n\r\nstatic int getOrdinalForReportType(ReportType t) {\r\n return (t == ReportType::Add);\r\n}\r\n\r\nclass ReportItem {\r\npublic:\r\n void * node;\r\n ReportType t;\r\n ReportItem * volatile next;\r\n int key;\r\n int id;\r\n \r\n ReportItem() {}\r\n void init(void * node, ReportType t, int key) {\r\n this->node = node;\r\n this->t = t;\r\n next = NULL;\r\n this->key = key;\r\n id = 0;\r\n }\r\n};\r\n\r\nclass CompactReportItem {\r\npublic:\r\n void * node;\r\n ReportType t;\r\n int key;\r\n int id;\r\n \r\n CompactReportItem() {}\r\n void init(void * node, ReportType t, int key) {\r\n this->node = node;\r\n this->t = t;\r\n this->key = key;\r\n id = 0;\r\n }\r\n};\r\n\r\nstruct {\r\n bool operator()(CompactReportItem * a, CompactReportItem * b) const {\r\n if (a->key != b->key) return a->key < b->key;\r\n if (a->node != b->node) return (long long) a->node < (long long) b->node;\r\n return getOrdinalForReportType(a->t) < getOrdinalForReportType(b->t);\r\n }\r\n} compareCRI;\r\n\r\n#endif /* REPORTITEM_H */\r\n\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/rq/snapcollector/snapcollector.h", "content": "/* \r\n * File: snapcollector.h\r\n * Author: trbot\r\n *\r\n * Created on June 21, 2017, 4:57 PM\r\n */\r\n\r\n#ifndef SNAPCOLLECTOR_H\r\n#define SNAPCOLLECTOR_H\r\n\r\n#include \r\n#include \r\n#include \r\n#include \"reportitem.h\"\r\n#include \r\n\r\ntemplate \r\nclass SnapCollector {\r\npublic:\r\n int NUM_THREADS;\r\n \r\n class NodeWrapper {\r\n public:\r\n NodeType * node;\r\n NodeWrapper * volatile next;\r\n K key;\r\n \r\n NodeWrapper() {}\r\n void init(K key) {\r\n this->node = NULL;\r\n this->next = NULL;\r\n this->key = key;\r\n }\r\n void init(NodeType * node, K key) {\r\n this->node = node;\r\n this->next = NULL;\r\n this->key = key;\r\n }\r\n };\r\n \r\nprivate:\r\n ReportItem * volatile * reportHeads;\r\n ReportItem * volatile * reportTails;\r\n \r\n NodeWrapper * volatile head;\r\n NodeWrapper * volatile tail;\r\n ReportItem * blocker;\r\n volatile bool active;\r\n \r\n // variables used for aggregating reports after they are collected\r\n void ** currLocations;\r\n int * currRepLocations;\r\n std::vector * volatile gAllReports;\r\n \r\n K KEY_MAX;\r\n K KEY_MIN;\r\n\r\nprivate:\r\n \r\n inline bool isBlocker(NodeWrapper const * const wrapper) {\r\n if (wrapper) {\r\n K key = wrapper->key;\r\n NodeType * node = wrapper->node;\r\n K key2 = KEY_MAX;\r\n return (key == key2 && node == NULL);\r\n }\r\n return false;\r\n }\r\n \r\n template \r\n inline void __retireAllReports(const int tid, std::vector * v, RecordManager * recmgr) {\r\n if (v == NULL) return;\r\n for (auto it = v->begin(); it != v->end(); it++) {\r\n // retire compact report items\r\n recmgr->retire(tid, *it);\r\n }\r\n }\r\n \r\n template \r\n inline void __deallocateAllReports(const int tid, std::vector * v, RecordManager * recmgr) {\r\n if (v == NULL) return;\r\n for (auto it = v->begin(); it != v->end(); it++) {\r\n // deallocate compact report items\r\n recmgr->deallocate(tid, *it);\r\n }\r\n delete v;\r\n }\r\n \r\npublic:\r\n \r\n template \r\n void init(const int tid, const int numProcesses, RecordManager * const recmgr, const K _KEY_MIN, const K _KEY_MAX) {\r\n this->KEY_MIN = _KEY_MIN;\r\n this->KEY_MAX = _KEY_MAX;\r\n \r\n this->NUM_THREADS = numProcesses;\r\n this->reportHeads = new ReportItem * volatile [NUM_THREADS*PREFETCH_SIZE_WORDS];\r\n this->reportTails = new ReportItem * volatile [NUM_THREADS*PREFETCH_SIZE_WORDS];\r\n // head = new NodeWrapper(std::numeric_limits::min())\r\n this->head = recmgr->template allocate(tid);\r\n#ifdef __HANDLE_STATS\r\n GSTATS_APPEND(tid, extra_type2_allocated_addresses, ((long long) head)%(1<<12));\r\n#endif\r\n this->head->init(this->KEY_MIN);\r\n this->tail = this->head;\r\n// oldTail = NULL;\r\n // blocker = new ReportItem(NULL, ReportType::Add, -1)\r\n this->blocker = recmgr->template allocate(tid);\r\n#ifdef __HANDLE_STATS\r\n GSTATS_APPEND(tid, extra_type3_allocated_addresses, ((long long) blocker)%(1<<12));\r\n#endif\r\n this->blocker->init(NULL, ReportType::Add, -1);\r\n this->active = true;\r\n this->currLocations = new void * [NUM_THREADS*PREFETCH_SIZE_WORDS];\r\n this->currRepLocations = new int[NUM_THREADS*PREFETCH_SIZE_WORDS];\r\n this->gAllReports = NULL;\r\n for (int i=0;ireportHeads[i*PREFETCH_SIZE_WORDS] = recmgr->template allocate(tid);\r\n#ifdef __HANDLE_STATS\r\n GSTATS_APPEND(tid, extra_type3_allocated_addresses, ((long long) reportHeads[i*PREFETCH_SIZE_WORDS])%(1<<12));\r\n#endif\r\n this->reportHeads[i*PREFETCH_SIZE_WORDS]->init(NULL, ReportType::Add, -1); // sentinel head.\r\n this->reportTails[i*PREFETCH_SIZE_WORDS] = this->reportHeads[i*PREFETCH_SIZE_WORDS];\r\n \r\n this->currLocations[i*PREFETCH_SIZE_WORDS] = NULL;\r\n this->currRepLocations[i*PREFETCH_SIZE_WORDS] = 0;\r\n }\r\n }\r\n \r\n ~SnapCollector() {\r\n if (reportHeads) delete[] reportHeads;\r\n if (reportTails) delete[] reportTails;\r\n if (currLocations) delete[] currLocations;\r\n if (currRepLocations) delete[] currRepLocations;\r\n if (gAllReports) delete gAllReports;\r\n }\r\n \r\n template \r\n void retire(const int tid, RecordManager * recmgr) {\r\n // retire report items\r\n for (int i=0;iretire(tid, curr); // blocker can exist in many per-thread lists, but we only want to retire it once, below.\r\n curr = curr->next;\r\n }\r\n }\r\n // retire blocker\r\n recmgr->retire(tid, blocker);\r\n // if a thread has changed tail to point to a \"blocker,\" then\r\n // threads may have appended node wrappers to the blocker,\r\n // so we have to retire any such node wrappers\r\n NodeWrapper * curr = this->tail;\r\n if (isBlocker(curr)) {\r\n while (curr != NULL) {\r\n recmgr->retire(tid, curr);\r\n curr = curr->next;\r\n }\r\n }\r\n // retire node wrappers\r\n curr = head;\r\n while (curr != NULL) { // && curr != tail /*&& curr != oldTail*/) {\r\n recmgr->retire(tid, curr);\r\n curr = curr->next;\r\n }\r\n // retire the contents of gAllReports\r\n __retireAllReports(tid, gAllReports, recmgr);\r\n // retire snap collector\r\n recmgr->retire(tid, this);\r\n }\r\n \r\n // TIMNAT: Implemented according to the optimization in A.3:\r\n // TIMNAT: Only accept nodes whose key is higher than the last, and return the last node. \r\n template \r\n NodeType * AddNode(const int tid, NodeType * node, K key, RecordManager * recmgr) {\r\n NodeWrapper * last = tail;\r\n if (last->key >= key) // TIMNAT: trying to add an out of place node.\r\n return last->node;\r\n\r\n // advance tail pointer if needed\r\n if (last->next != NULL) {\r\n if (last == tail) __sync_bool_compare_and_swap(&tail, last, last->next);\r\n return tail->node;\r\n }\r\n \r\n NodeWrapper * newNode = recmgr->template allocate(tid);\r\n#ifdef __HANDLE_STATS\r\n GSTATS_APPEND(tid, extra_type2_allocated_addresses, ((long long) newNode)%(1<<12));\r\n#endif\r\n newNode->init(node, key);\r\n if (__sync_bool_compare_and_swap(&last->next, NULL, newNode)) {\r\n __sync_bool_compare_and_swap(&tail, last, newNode);\r\n return node;\r\n } else {\r\n recmgr->deallocate(tid, newNode);\r\n return tail->node;\r\n }\r\n }\r\n \r\n template \r\n void Report(int tid, NodeType * Node, ReportType t, K key, RecordManager * recmgr) {\r\n ReportItem * reportTail = reportTails[tid*PREFETCH_SIZE_WORDS];\r\n ReportItem * newItem = recmgr->template allocate(tid);\r\n#ifdef __HANDLE_STATS\r\n GSTATS_APPEND(tid, extra_type3_allocated_addresses, ((long long) newItem)%(1<<12));\r\n#endif\r\n newItem->init(Node, t, key);\r\n if (__sync_bool_compare_and_swap(&reportTail->next, NULL, newItem)) {\r\n reportTails[tid*PREFETCH_SIZE_WORDS] = newItem;\r\n } else {\r\n recmgr->deallocate(tid, newItem);\r\n }\r\n }\r\n \r\n bool IsActive() {\r\n// __sync_synchronize();\r\n// SOFTWARE_BARRIER;\r\n bool result = active;\r\n// SOFTWARE_BARRIER;\r\n return result;\r\n }\r\n \r\n template \r\n void BlockFurtherPointers(const int tid, RecordManager * recmgr) {\r\n NodeWrapper * blocker = recmgr->template allocate(tid);\r\n#ifdef __HANDLE_STATS\r\n GSTATS_APPEND(tid, extra_type2_allocated_addresses, ((long long) blocker)%(1<<12));\r\n#endif\r\n blocker->init(NULL, KEY_MAX);\r\n \r\n#if 1\r\n while (true) {\r\n NodeWrapper * old = this->tail;\r\n if (isBlocker(old)) { // old is a blocker, so no need to add our own blocker\r\n recmgr->deallocate(tid, blocker);\r\n return;\r\n }\r\n if (__sync_bool_compare_and_swap(&this->tail, old, blocker)) {\r\n return;\r\n }\r\n }\r\n#else\r\n tail = blocker;\r\n#endif\r\n }\r\n \r\n /**\r\n * note: the parameters are used for the timestamping mechanism of the\r\n * test harness. they are NOT inherently needed by the snap collector.\r\n */\r\n void Deactivate(pthread_rwlock_t * const rwlock, volatile long long * timestamp, long long * rq_lin_time) {\r\n#ifdef RQ_USE_TIMESTAMPS\r\n if (pthread_rwlock_wrlock(rwlock)) error(\"could not write-lock rwlock\");\r\n active = false; // range query is linearized here\r\n *timestamp = *timestamp + 1;\r\n *rq_lin_time = *timestamp; //++(*timestamp);\r\n //std::cout<<\"timestamp=\"<<*timestamp<next == NULL)\r\n __sync_bool_compare_and_swap(&reportTail->next, NULL, blocker);\r\n // assert cas succeeded OR reportTail->next == blocker\r\n }\r\n }\r\n\r\nprivate:\r\n \r\n // TIMNAT: What follows is functions that are used to work with the snapshot while it is\r\n // TIMNAT: already taken. These functions are used to iterate over the nodes of the snapshot.\r\n\r\n template \r\n void AddReports(const int tid, std::vector * allReports, ReportItem * curr, RecordManager * recmgr) {\r\n curr = curr->next;\r\n while (curr != NULL && curr != blocker) {\r\n CompactReportItem * newItem = recmgr->template allocate(tid);\r\n#ifdef __HANDLE_STATS\r\n GSTATS_APPEND(tid, extra_type4_allocated_addresses, ((long long) newItem)%(1<<12));\r\n#endif\r\n newItem->init(curr->node, curr->t, curr->key);\r\n allReports->push_back(newItem);\r\n curr = curr->next;\r\n }\r\n }\r\n\r\npublic:\r\n // An optimization: sort the reports and nodes.\r\n template \r\n void Prepare(int tid, RecordManager * recmgr) {\r\n currLocations[tid*PREFETCH_SIZE_WORDS] = head;\r\n currRepLocations[tid*PREFETCH_SIZE_WORDS] = 0;\r\n if (gAllReports != NULL) return;\r\n\r\n std::vector * allReports = new std::vector();\r\n for (int i = 0; i < NUM_THREADS; i++) {\r\n AddReports(tid, allReports, reportHeads[i*PREFETCH_SIZE_WORDS], recmgr);\r\n if (gAllReports != NULL) {\r\n // failed to publish allReports -- clean it up\r\n __deallocateAllReports(tid, allReports, recmgr);\r\n return;\r\n }\r\n }\r\n assert(!active);\r\n#ifdef SNAPCOLLECTOR_PRINT_RQS\r\n std::cout<<\"this=\"<<(long long) this<<\" allReports size=\"<size()<begin(), allReports->end(), compareCRI);\r\n if (__sync_bool_compare_and_swap(&gAllReports, NULL, allReports)) {\r\n // published allReports\r\n } else {\r\n // failed to publish allReports -- clean it up\r\n __deallocateAllReports(tid, allReports, recmgr);\r\n }\r\n }\r\n \r\n NodeType * GetNext(int tid) {\r\n NodeWrapper * currLoc = (NodeWrapper *) currLocations[tid*PREFETCH_SIZE_WORDS];\r\n int currRepLoc = currRepLocations[tid*PREFETCH_SIZE_WORDS];\r\n std::vector * allReports = gAllReports;\r\n\r\n while (true) {\r\n CompactReportItem * rep = NULL;\r\n K repKey = KEY_MAX;\r\n if (allReports->size() > currRepLoc) {\r\n rep = (*allReports)[currRepLoc];\r\n repKey = rep->key;\r\n }\r\n K nodeKey = KEY_MAX;\r\n NodeWrapper * next = currLoc->next;\r\n if (next != NULL) {\r\n nodeKey = next->key;\r\n }\r\n\r\n // Option 1: node key < rep key. Return node.\r\n if (nodeKey < repKey) {\r\n currLocations[tid*PREFETCH_SIZE_WORDS] = next;\r\n currRepLocations[tid*PREFETCH_SIZE_WORDS] = currRepLoc;\r\n return next->node;\r\n }\r\n\r\n // Option 2: node key == rep key \r\n if (nodeKey == repKey) {\r\n // 2.a - both are infinity - iteration done.\r\n if (nodeKey == KEY_MAX) {\r\n currLocations[tid*PREFETCH_SIZE_WORDS] = currLoc;\r\n currRepLocations[tid*PREFETCH_SIZE_WORDS] = currRepLoc;\r\n return NULL;\r\n }\r\n // node and report with the same key ::\r\n\r\n // skip not-needed reports\r\n while (currRepLoc + 1 < allReports->size()) {\r\n CompactReportItem * nextRep = (*allReports)[currRepLoc + 1];\r\n // dismiss a duplicate, or an insert followed by a matching delete:\r\n if (rep->key == nextRep->key && rep->node == nextRep->node) {\r\n currRepLoc++;\r\n rep = nextRep;\r\n } else {\r\n break;\r\n }\r\n }\r\n // standing on an insert report to a node I am holding:\r\n // 1. Return the current node.\r\n // 2. Skip over rest of reports for that key.\r\n if (rep->t == ReportType::Add && (NodeType *) rep->node == next->node) {\r\n while (currRepLoc < allReports->size()\r\n && (*allReports)[currRepLoc]->key == rep->key) {\r\n currRepLoc++;\r\n }\r\n currRepLocations[tid*PREFETCH_SIZE_WORDS] = currRepLoc;\r\n currLocations[tid*PREFETCH_SIZE_WORDS] = next;\r\n return next->node;\r\n }\r\n // standing on an insert report to a different node than I hold:\r\n // 1. Return the reported node.\r\n // 2. Skip over rest of reports for that key.\r\n if (rep->t == ReportType::Add && (NodeType *) rep->node != next->node) {\r\n NodeType * returnValue = (NodeType *) rep->node;\r\n while (currRepLoc < allReports->size()\r\n && (*allReports)[currRepLoc]->key == rep->key) {\r\n currRepLoc++;\r\n }\r\n currRepLocations[tid*PREFETCH_SIZE_WORDS] = currRepLoc;\r\n currLocations[tid*PREFETCH_SIZE_WORDS] = next;\r\n return returnValue;\r\n }\r\n // standing on a delete report to a different node than I hold:\r\n // skip over it and continue the big loop.\r\n if (rep->t == ReportType::Remove && (NodeType *) rep->node != next->node) {\r\n currRepLoc++;\r\n continue;\r\n }\r\n // standing on a delete report to the node that I hold:\r\n // 1. advance over the node that I hold.\r\n // 2. advance with the report.\r\n // 3. continue the bigloop\r\n currLoc = next;\r\n currRepLoc++;\r\n continue;\r\n }\r\n\r\n // Option 3: node key > rep key\r\n if (nodeKey > repKey) {\r\n // skip not-needed reports\r\n while (currRepLoc + 1 < allReports->size()) {\r\n CompactReportItem * nextRep = (*allReports)[currRepLoc + 1];\r\n // dismiss a duplicate, or an insert followed by a matching delete:\r\n if (rep->key == nextRep->key && rep->node == nextRep->node) {\r\n currRepLoc++;\r\n rep = nextRep;\r\n } else {\r\n break;\r\n }\r\n }\r\n // a delete report - skip over it.\r\n if (rep->t == ReportType::Remove) {\r\n currRepLoc++;\r\n continue;\r\n }\r\n\r\n // an insert report:\r\n // 1. skip over rest of the reports for the same key.\r\n // 2. return the node.\r\n if (rep->t == ReportType::Add) {\r\n NodeType * returnValue = (NodeType *) rep->node;\r\n while (currRepLoc < allReports->size()\r\n && (*allReports)[currRepLoc]->key == rep->key) {\r\n currRepLoc++;\r\n }\r\n currRepLocations[tid*PREFETCH_SIZE_WORDS] = currRepLoc;\r\n currLocations[tid*PREFETCH_SIZE_WORDS] = currLoc;\r\n return returnValue;\r\n }\r\n }\r\n }\r\n }\r\n\r\n};\r\n\r\n#endif /* SNAPCOLLECTOR_H */\r\n\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/rq/snapcollector/snapcollector_test.cpp", "content": "/* \r\n * File: test.cpp\r\n * Author: trbot\r\n *\r\n * Created on June 21, 2017, 5:25 PM\r\n */\r\n\r\n#include \r\n#include \r\n#include \r\n#include \"snapcollector.h\"\r\n#include \"rq_snapcollector.h\"\r\n\r\nusing namespace std;\r\n\r\nclass Node {\r\npublic:\r\n int key;\r\n \r\n volatile bool marked;\r\n volatile long long itime;\r\n volatile long long dtime;\r\n \r\n Node(int key) : key(key) {}\r\n};\r\n\r\nclass DataStructure {\r\npublic:\r\n inline bool isLogicallyDeleted(const int tid, Node * node) {\r\n return node->marked;\r\n }\r\n \r\n inline int getKeys(const int tid, Node * node, int * const outputKeys) {\r\n outputKeys[0] = node->key;\r\n return 1;\r\n }\r\n \r\n bool isInRange(const int& key, const int& lo, const int& hi) {\r\n return lo <= key && key <= hi;\r\n }\r\n};\r\n\r\n/*\r\n * \r\n */\r\nint main(int argc, char** argv) {\r\n DataStructure ds;\r\n Node node (17);\r\n const int numProcessors = 1;\r\n SnapCollector sc (numProcessors);\r\n sc.AddNode(&node, node.key);\r\n \r\n RQProvider prov (numProcessors, &ds);\r\n Node * inserted[] = {NULL};\r\n Node * deleted[] = {NULL};\r\n prov.linearize_update_at_cas(1, &node.key, 17, 18, inserted, deleted, (void *) NULL);\r\n \r\n return 0;\r\n}\r\n\r\n" }, { "path": "datastructures/trevor_brown_abtree/common/rwlock.h", "content": "/* \r\n * File: rwlock.h\r\n * Author: trbot\r\n *\r\n * Created on June 29, 2017, 8:25 PM\r\n */\r\n\r\n#ifndef RWLOCK_H\r\n#define RWLOCK_H\r\n\r\n#ifdef RWLOCK_PTHREADS\r\n#elif defined RWLOCK_FAVOR_WRITERS\r\n#elif defined RWLOCK_FAVOR_READERS\r\n#else\r\n// #warning \"No RWLOCK implementation specified... using default: favour READERS. See rwlock.h for options. Note that this setting only affects algorithms that use the lock-based range query provider in common/rq/rq_rwlock.h.\"\r\n #define RWLOCK_FAVOR_READERS\r\n// #error Must specify RWLOCK implementation; see rwlock.h\r\n#endif\r\n\r\n#ifdef RWLOCK_PTHREADS\r\n\r\nclass RWLock {\r\nprivate:\r\n pthread_rwlock_t lock;\r\n \r\npublic:\r\n RWLock() {\r\n if (pthread_rwlock_init(&lock, NULL)) error(\"could not init rwlock\");\r\n }\r\n ~RWLock() {\r\n if (pthread_rwlock_destroy(&lock)) error(\"could not destroy rwlock\");\r\n }\r\n inline void readLock() {\r\n if (pthread_rwlock_rdlock(&lock)) error(\"could not read-lock rwlock\");\r\n }\r\n inline void readUnlock() {\r\n if (pthread_rwlock_unlock(&lock)) error(\"could not read-unlock rwlock\");\r\n }\r\n inline void writeLock() {\r\n if (pthread_rwlock_wrlock(&lock)) error(\"could not write-lock rwlock\");\r\n }\r\n inline void writeUnlock() {\r\n if (pthread_rwlock_unlock(&lock)) error(\"could not write-unlock rwlock\");\r\n }\r\n inline bool isWriteLocked() {\r\n cout<<\"ERROR: isWriteLocked() is not implemented\"<.\r\n */\r\n\r\n#ifndef ABTREE_H\r\n#define\tABTREE_H\r\n\r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \"record_manager.h\"\r\n#include \"descriptors.h\"\r\n#include \"rq_provider.h\"\r\n\r\nnamespace abtree_ns {\r\n\r\n #ifndef TRACE\r\n #define TRACE if(0)\r\n #endif\r\n #ifndef DEBUG\r\n #define DEBUG if(0)\r\n #endif\r\n #ifndef DEBUG1\r\n #define DEBUG1 if(0)\r\n #endif\r\n #ifndef DEBUG2\r\n #define DEBUG2 if(0)\r\n #endif\r\n\r\n #define ABTREE_ENABLE_DESTRUCTOR\r\n\r\n using namespace std;\r\n \r\n template \r\n struct Node;\r\n \r\n template \r\n struct SCXRecord;\r\n \r\n template \r\n class wrapper_info {\r\n public:\r\n const static int MAX_NODES = DEGREE+2;\r\n Node * nodes[MAX_NODES];\r\n SCXRecord * scxPtrs[MAX_NODES];\r\n Node * newNode;\r\n Node * volatile * field;\r\n int state;\r\n char numberOfNodes;\r\n char numberOfNodesToFreeze;\r\n char numberOfNodesAllocated;\r\n\r\n // for rqProvider\r\n Node * insertedNodes[MAX_NODES+1];\r\n Node * deletedNodes[MAX_NODES+1];\r\n };\r\n\r\n template \r\n struct SCXRecord {\r\n const static int STATE_INPROGRESS = 0;\r\n const static int STATE_COMMITTED = 1;\r\n const static int STATE_ABORTED = 2;\r\n union {\r\n struct {\r\n volatile mutables_t mutables;\r\n\r\n int numberOfNodes;\r\n int numberOfNodesToFreeze;\r\n\r\n Node * newNode;\r\n Node * volatile * field;\r\n Node * nodes[wrapper_info::MAX_NODES]; // array of pointers to nodes\r\n SCXRecord * scxPtrsSeen[wrapper_info::MAX_NODES]; // array of pointers to scx records\r\n\r\n // for rqProvider\r\n Node * insertedNodes[wrapper_info::MAX_NODES+1];\r\n Node * deletedNodes[wrapper_info::MAX_NODES+1];\r\n } __attribute__((packed)) c; // WARNING: be careful with atomicity because of packed attribute!!! (this means no atomic vars smaller than word size, and all atomic vars must start on a word boundary when fields are packed tightly)\r\n char bytes[2*PREFETCH_SIZE_BYTES];\r\n };\r\n const static int size = sizeof(c);\r\n };\r\n \r\n template \r\n struct Node {\r\n SCXRecord * volatile scxPtr;\r\n int leaf; // 0 or 1\r\n volatile int marked; // 0 or 1\r\n int weight; // 0 or 1\r\n int size; // degree of node\r\n K searchKey;\r\n#if defined(RQ_LOCKFREE) || defined(RQ_RWLOCK) || defined(HTM_RQ_RWLOCK)\r\n volatile long long itime; // for use by range query algorithm\r\n volatile long long dtime; // for use by range query algorithm\r\n#endif\r\n K keys[DEGREE];\r\n Node * volatile ptrs[DEGREE];\r\n\r\n inline bool isLeaf() {\r\n return leaf;\r\n }\r\n inline int getKeyCount() {\r\n return isLeaf() ? size : size-1;\r\n }\r\n inline int getABDegree() {\r\n return size;\r\n }\r\n template \r\n inline int getChildIndex(const K& key, Compare cmp) {\r\n int nkeys = getKeyCount();\r\n int retval = 0;\r\n while (retval < nkeys && !cmp(key, (const K&) keys[retval])) {\r\n ++retval;\r\n }\r\n return retval;\r\n }\r\n template \r\n inline int getKeyIndex(const K& key, Compare cmp) {\r\n int nkeys = getKeyCount();\r\n int retval = 0;\r\n while (retval < nkeys && cmp((const K&) keys[retval], key)) {\r\n ++retval;\r\n }\r\n return retval;\r\n }\r\n };\r\n\r\n template \r\n class abtree {\r\n\r\n // the following bool determines whether the optimization to guarantee\r\n // amortized constant rebalancing (at the cost of decreasing average degree\r\n // by at most one) is used.\r\n // if it is false, then an amortized logarithmic number of rebalancing steps\r\n // may be performed per operation, but average degree increases slightly.\r\n char padding0[PREFETCH_SIZE_BYTES];\r\n const bool ALLOW_ONE_EXTRA_SLACK_PER_NODE;\r\n\r\n const int b;\r\n const int a;\r\n\r\n RecManager * const recordmgr;\r\n RQProvider, abtree, RecManager, false, false> * const rqProvider;\r\n char padding1[PREFETCH_SIZE_BYTES];\r\n Compare cmp;\r\n\r\n // descriptor reduction algorithm\r\n #ifndef comma\r\n #define comma ,\r\n #endif\r\n #define DESC1_ARRAY records\r\n #define DESC1_T SCXRecord\r\n #define MUTABLES1_OFFSET_ALLFROZEN 0\r\n #define MUTABLES1_OFFSET_STATE 1\r\n #define MUTABLES1_MASK_ALLFROZEN 0x1\r\n #define MUTABLES1_MASK_STATE 0x6\r\n #define MUTABLES1_NEW(mutables) \\\r\n ((((mutables)&MASK1_SEQ)+(1<::STATE_INPROGRESS<::STATE_COMMITTED< * entry;\r\n char padding3[PREFETCH_SIZE_BYTES];\r\n\r\n #define DUMMY ((SCXRecord*) (void*) TAGPTR1_STATIC_DESC(0))\r\n #define FINALIZED ((SCXRecord*) (void*) TAGPTR1_DUMMY_DESC(1))\r\n #define FAILED ((SCXRecord*) (void*) TAGPTR1_DUMMY_DESC(2))\r\n\r\n #define arraycopy(src, srcStart, dest, destStart, len) \\\r\n for (int ___i=0;___i<(len);++___i) { \\\r\n (dest)[(destStart)+___i] = (src)[(srcStart)+___i]; \\\r\n }\r\n #define arraycopy_ptrs(src, srcStart, dest, destStart, len) \\\r\n for (int ___i=0;___i<(len);++___i) { \\\r\n rqProvider->write_addr(tid, &(dest)[(destStart)+___i], \\\r\n rqProvider->read_addr(tid, &(src)[(srcStart)+___i])); \\\r\n }\r\n\r\n private:\r\n void* doInsert(const int tid, const K& key, void * const value, const bool replace);\r\n\r\n // returns true if the invocation of this method\r\n // (and not another invocation of a method performed by this method)\r\n // performed an scx, and false otherwise\r\n bool fixWeightViolation(const int tid, Node* viol);\r\n\r\n // returns true if the invocation of this method\r\n // (and not another invocation of a method performed by this method)\r\n // performed an scx, and false otherwise\r\n bool fixDegreeViolation(const int tid, Node* viol);\r\n\r\n bool llx(const int tid, Node* r, Node ** snapshot, const int i, SCXRecord ** ops, Node ** nodes);\r\n SCXRecord* llx(const int tid, Node* r, Node ** snapshot);\r\n bool scx(const int tid, wrapper_info * info);\r\n void helpOther(const int tid, tagptr_t tagptr);\r\n int help(const int tid, const tagptr_t tagptr, SCXRecord const * const snap, const bool helpingOther);\r\n\r\n SCXRecord* createSCXRecord(const int tid, wrapper_info * info);\r\n Node* allocateNode(const int tid);\r\n\r\n void freeSubtree(Node* node, int* nodes) {\r\n const int tid = 0;\r\n if (node == NULL) return;\r\n if (!node->isLeaf()) {\r\n for (int i=0;igetABDegree();++i) {\r\n freeSubtree(node->ptrs[i], nodes);\r\n }\r\n }\r\n ++(*nodes);\r\n recordmgr->retire(tid, node);\r\n }\r\n\r\n int init[MAX_TID_POW2] = {0,};\r\npublic:\r\n void * const NO_VALUE;\r\n const int NUM_PROCESSES;\r\n #ifdef USE_DEBUGCOUNTERS\r\n debugCounters * const counters; // debug info\r\n #endif\r\n\r\n /**\r\n * This function must be called once by each thread that will\r\n * invoke any functions on this class.\r\n * \r\n * It must be okay that we do this with the main thread and later with another thread!\r\n */\r\n void initThread(const int tid) {\r\n if (init[tid]) return; else init[tid] = !init[tid];\r\n \r\n recordmgr->initThread(tid);\r\n rqProvider->initThread(tid);\r\n }\r\n void deinitThread(const int tid) {\r\n if (!init[tid]) return; else init[tid] = !init[tid];\r\n\r\n rqProvider->deinitThread(tid);\r\n recordmgr->deinitThread(tid);\r\n }\r\n\r\n /**\r\n * Creates a new relaxed (a,b)-tree wherein:
\r\n * each internal node has up to DEGREE child pointers, and
\r\n * each leaf has up to DEGREE key/value pairs, and
\r\n * keys are ordered according to the provided comparator.\r\n */\r\n abtree(const int numProcesses, \r\n const K anyKey,\r\n int suspectedCrashSignal = SIGQUIT)\r\n : ALLOW_ONE_EXTRA_SLACK_PER_NODE(true)\r\n , b(DEGREE)\r\n , a(DEGREE/2 - 2)\r\n , recordmgr(new RecManager(numProcesses, suspectedCrashSignal))\r\n , rqProvider(new RQProvider, abtree, RecManager, false, false>(numProcesses, this, recordmgr))\r\n , NO_VALUE((void *) -1LL)\r\n , NUM_PROCESSES(numProcesses) \r\n {\r\n cmp = Compare();\r\n \r\n const int tid = 0;\r\n initThread(tid);\r\n\r\n recordmgr->enterQuiescentState(tid);\r\n \r\n DESC1_INIT_ALL(numProcesses);\r\n\r\n SCXRecord *dummy = TAGPTR1_UNPACK_PTR(DUMMY);\r\n dummy->c.mutables = MUTABLES1_INIT_DUMMY;\r\n TRACE COUTATOMICTID(\"DUMMY mutables=\"<c.mutables<* _entryLeft = allocateNode(tid);\r\n _entryLeft->scxPtr = DUMMY;\r\n _entryLeft->leaf = true;\r\n _entryLeft->marked = false;\r\n _entryLeft->weight = true;\r\n _entryLeft->size = 0;\r\n _entryLeft->searchKey = anyKey;\r\n\r\n Node* _entry = allocateNode(tid);\r\n _entry = allocateNode(tid);\r\n _entry->scxPtr = DUMMY;\r\n _entry->leaf = false;\r\n _entry->marked = false;\r\n _entry->weight = true;\r\n _entry->size = 1;\r\n _entry->searchKey = anyKey;\r\n _entry->ptrs[0] = _entryLeft;\r\n \r\n // need to simulate real insertion of root and the root's child,\r\n // since range queries will actually try to add these nodes,\r\n // and we don't want blocking rq providers to spin forever\r\n // waiting for their itimes to be set to a positive number.\r\n Node* insertedNodes[] = {_entry, _entryLeft, NULL};\r\n Node* deletedNodes[] = {NULL};\r\n rqProvider->linearize_update_at_write(tid, &entry, _entry, insertedNodes, deletedNodes);\r\n }\r\n\r\n #ifdef ABTREE_ENABLE_DESTRUCTOR \r\n ~abtree() {\r\n int nodes = 0;\r\n freeSubtree(entry, &nodes);\r\n// COUTATOMIC(\"main thread: deleted tree containing \"<printStatus();\r\n delete recordmgr;\r\n }\r\n #endif\r\n\r\n Node * debug_getEntryPoint() { return entry; }\r\n\r\n private:\r\n /*******************************************************************\r\n * Utility functions for integration with the test harness\r\n *******************************************************************/\r\n\r\n int sequentialSize(Node* node) {\r\n if (node->isLeaf()) {\r\n return node->getKeyCount();\r\n }\r\n int retval = 0;\r\n for (int i=0;igetABDegree();++i) {\r\n Node* child = node->ptrs[i];\r\n retval += sequentialSize(child);\r\n }\r\n return retval;\r\n }\r\n int sequentialSize() {\r\n return sequentialSize(entry->ptrs[0]);\r\n }\r\n\r\n int getNumberOfLeaves(Node* node) {\r\n if (node == NULL) return 0;\r\n if (node->isLeaf()) return 1;\r\n int result = 0;\r\n for (int i=0;igetABDegree();++i) {\r\n result += getNumberOfLeaves(node->ptrs[i]);\r\n }\r\n return result;\r\n }\r\n const int getNumberOfLeaves() {\r\n return getNumberOfLeaves(entry->ptrs[0]);\r\n }\r\n int getNumberOfInternals(Node* node) {\r\n if (node == NULL) return 0;\r\n if (node->isLeaf()) return 0;\r\n int result = 1;\r\n for (int i=0;igetABDegree();++i) {\r\n result += getNumberOfInternals(node->ptrs[i]);\r\n }\r\n return result;\r\n }\r\n const int getNumberOfInternals() {\r\n return getNumberOfInternals(entry->ptrs[0]);\r\n }\r\n const int getNumberOfNodes() {\r\n return getNumberOfLeaves() + getNumberOfInternals();\r\n }\r\n\r\n int getSumOfKeyDepths(Node* node, int depth) {\r\n if (node == NULL) return 0;\r\n if (node->isLeaf()) return depth * node->getKeyCount();\r\n int result = 0;\r\n for (int i=0;igetABDegree();i++) {\r\n result += getSumOfKeyDepths(node->ptrs[i], 1+depth);\r\n }\r\n return result;\r\n }\r\n const int getSumOfKeyDepths() {\r\n return getSumOfKeyDepths(entry->ptrs[0], 0);\r\n }\r\n const double getAverageKeyDepth() {\r\n long sz = sequentialSize();\r\n return (sz == 0) ? 0 : getSumOfKeyDepths() / sz;\r\n }\r\n\r\n int getHeight(Node* node, int depth) {\r\n if (node == NULL) return 0;\r\n if (node->isLeaf()) return 0;\r\n int result = 0;\r\n for (int i=0;igetABDegree();i++) {\r\n int retval = getHeight(node->ptrs[i], 1+depth);\r\n if (retval > result) result = retval;\r\n }\r\n return result+1;\r\n }\r\n const int getHeight() {\r\n return getHeight(entry->ptrs[0], 0);\r\n }\r\n\r\n int getKeyCount(Node* entry) {\r\n if (entry == NULL) return 0;\r\n if (entry->isLeaf()) return entry->getKeyCount();\r\n int sum = 0;\r\n for (int i=0;igetABDegree();++i) {\r\n sum += getKeyCount(entry->ptrs[i]);\r\n }\r\n return sum;\r\n }\r\n int getTotalDegree(Node* entry) {\r\n if (entry == NULL) return 0;\r\n int sum = entry->getKeyCount();\r\n if (entry->isLeaf()) return sum;\r\n for (int i=0;igetABDegree();++i) {\r\n sum += getTotalDegree(entry->ptrs[i]);\r\n }\r\n return 1+sum; // one more children than keys\r\n }\r\n int getNodeCount(Node* entry) {\r\n if (entry == NULL) return 0;\r\n if (entry->isLeaf()) return 1;\r\n int sum = 1;\r\n for (int i=0;igetABDegree();++i) {\r\n sum += getNodeCount(entry->ptrs[i]);\r\n }\r\n return sum;\r\n }\r\n double getAverageDegree() {\r\n return getTotalDegree(entry) / (double) getNodeCount(entry);\r\n }\r\n double getSpacePerKey() {\r\n return getNodeCount(entry)*2*b / (double) getKeyCount(entry);\r\n }\r\n\r\n long long getSumOfKeys(Node* node) {\r\n TRACE COUTATOMIC(\" getSumOfKeys(\"<isLeaf()<isLeaf()) {\r\n TRACE COUTATOMIC(\" leaf sum +=\");\r\n for (int i=0;igetKeyCount();++i) {\r\n sum += (long long) node->keys[i];\r\n TRACE COUTATOMIC(node->keys[i]);\r\n }\r\n TRACE COUTATOMIC(endl);\r\n } else {\r\n for (int i=0;igetABDegree();++i) {\r\n sum += getSumOfKeys(node->ptrs[i]);\r\n }\r\n }\r\n TRACE COUTATOMIC(\" getSumOfKeys(\"< erase(const int tid, const K& key);\r\n const pair find(const int tid, const K& key);\r\n bool contains(const int tid, const K& key);\r\n int rangeQuery(const int tid, const K& low, const K& hi, K * const resultKeys, void ** const resultValues);\r\n bool validate(const long long keysum, const bool checkkeysum) {\r\n if (checkkeysum) {\r\n long long treekeysum = getSumOfKeys();\r\n if (treekeysum != keysum) {\r\n cerr<<\"ERROR: tree keysum \"< * node) {\r\n return false;\r\n }\r\n\r\n inline int getKeys(const int tid, Node * node, K * const outputKeys, void ** const outputValues) {\r\n if (node->isLeaf()) {\r\n // leaf ==> its keys are in the set.\r\n const int sz = node->getKeyCount();\r\n for (int i=0;ikeys[i];\r\n outputValues[i] = (void *) node->ptrs[i];\r\n }\r\n return sz;\r\n }\r\n // note: internal ==> its keys are NOT in the set\r\n return 0;\r\n }\r\n\r\n bool isInRange(const K& key, const K& lo, const K& hi) {\r\n return (!cmp(key, lo) && !cmp(hi, key));\r\n }\r\n\r\n /**\r\n * END FUNCTIONS FOR RANGE QUERY SUPPORT\r\n */\r\n\r\n long long getSizeInNodes() {\r\n return getNumberOfNodes();\r\n }\r\n string getSizeString() {\r\n stringstream ss;\r\n int preallocated = wrapper_info::MAX_NODES * recordmgr->NUM_PROCESSES;\r\n ss< * node) {\r\n return sequentialSize(node);\r\n }\r\n long long getSize() {\r\n return sequentialSize();\r\n }\r\n RecManager * const debugGetRecMgr() {\r\n return recordmgr;\r\n }\r\n long long debugKeySum() {\r\n return getSumOfKeys();\r\n }\r\n };\r\n} // namespace\r\n\r\n#endif\t/* ABTREE_H */\r\n\r\n" }, { "path": "datastructures/trevor_brown_abtree/ds/brown_ext_abtree_lf/brown_ext_abtree_lf_adapter.h", "content": "/* \r\n * File: bst_adapter.h\r\n * Author: trbot\r\n *\r\n * Created on August 31, 2017, 6:53 PM\r\n */\r\n\r\n#ifndef BST_ADAPTER_H\r\n#define BST_ADAPTER_H\r\n\r\n#include \r\n#include \"brown_ext_abtree_lf_impl.h\"\r\n#include \"errors.h\"\r\nusing namespace abtree_ns;\r\n\r\n#define RECORD_MANAGER_T record_manager>\r\n#define DATA_STRUCTURE_T abtree, RECORD_MANAGER_T>\r\n\r\ntemplate , class Alloc = allocator_new, class Pool = pool_none>\r\nclass ds_adapter {\r\nprivate:\r\n const void * NO_VALUE;\r\n DATA_STRUCTURE_T * const ds;\r\n\r\npublic:\r\n ds_adapter(const int numThreads, const K ANY_KEY)\r\n : ds(new DATA_STRUCTURE_T(numThreads, ANY_KEY))\r\n {}\r\n ~ds_adapter() {\r\n delete ds;\r\n }\r\n \r\n void * getNoValue() {\r\n return ds->NO_VALUE;\r\n }\r\n \r\n void initThread(const int tid) {\r\n ds->initThread(tid);\r\n }\r\n void deinitThread(const int tid) {\r\n ds->deinitThread(tid);\r\n }\r\n\r\n bool contains(const int tid, const K& key) {\r\n return ds->contains(tid, key);\r\n }\r\n void * const insert(const int tid, const K& key, void * const val) {\r\n return ds->insert(tid, key, val);\r\n }\r\n void * const insertIfAbsent(const int tid, const K& key, void * const val) {\r\n return ds->insertIfAbsent(tid, key, val);\r\n }\r\n void * const erase(const int tid, const K& key) {\r\n return ds->erase(tid, key).first;\r\n }\r\n void * find(const int tid, const K& key) {\r\n return ds->find(tid, key).first;\r\n }\r\n int rangeQuery(const int tid, const K& lo, const K& hi, K * const resultKeys, void ** const resultValues) {\r\n return ds->rangeQuery(tid, lo, hi, resultKeys, resultValues);\r\n }\r\n /**\r\n * Sequential operation to get the number of keys in the set\r\n */\r\n int getSize() {\r\n return ds->getSize();\r\n }\r\n void printSummary() {\r\n stringstream ss;\r\n ss<getSizeInNodes()<<\" nodes in tree\";\r\n cout<debugGetRecMgr();\r\n recmgr->printStatus();\r\n }\r\n long long getKeyChecksum() {\r\n return ds->debugKeySum();\r\n }\r\n bool validateStructure() {\r\n return true;\r\n }\r\n void printObjectSizes() {\r\n std::cout<<\"sizes: node=\"\r\n <<(sizeof(Node))\r\n <<\" descriptor=\"<<(sizeof(SCXRecord))<<\" (statically allocated)\"\r\n <.\r\n */\r\n\r\n/**\r\n * Implementation note:\r\n * The ptrs arrays of internal nodes may be modified by calls to\r\n * rqProvider->linearize_update_at_cas or ->linearize_update_at_write.\r\n * Consequently, we must access access entries in the ptrs arrays of INTERNAL\r\n * nodes by performing calls to read_addr and write_addr (and linearize_...).\r\n * \r\n * However, the ptrs arrays of leaves represent fundamentally different data:\r\n * specifically values, or pointers to values, and NOT pointers to nodes.\r\n * Thus, the ptrs arrays of leaves CANNOT be modified by such calls.\r\n * So, we do NOT use these functions to access entries in leaves' ptrs arrays.\r\n */\r\n\r\n#ifndef ABTREE_IMPL_H\r\n#define\tABTREE_IMPL_H\r\n\r\n#include \"brown_ext_abtree_lf.h\"\r\n\r\n#define eassert(x, y) if ((x) != (y)) { cout<<\"ERROR: \"<<#x<<\" != \"<<#y<<\" :: \"<<#x<<\"=\"<\r\nabtree_ns::SCXRecord * abtree_ns::abtree::createSCXRecord(const int tid, wrapper_info * info) {\r\n \r\n SCXRecord * result = DESC1_NEW(tid);\r\n result->c.newNode = info->newNode;\r\n for (int i=0;inumberOfNodes;++i) {\r\n result->c.nodes[i] = info->nodes[i];\r\n }\r\n for (int i=0;inumberOfNodesToFreeze;++i) {\r\n result->c.scxPtrsSeen[i] = info->scxPtrs[i];\r\n }\r\n \r\n int i;\r\n for (i=0;info->insertedNodes[i];++i) result->c.insertedNodes[i] = info->insertedNodes[i];\r\n result->c.insertedNodes[i] = NULL;\r\n for (i=0;info->deletedNodes[i];++i) result->c.deletedNodes[i] = info->deletedNodes[i];\r\n result->c.deletedNodes[i] = NULL;\r\n \r\n result->c.field = info->field;\r\n result->c.numberOfNodes = info->numberOfNodes;\r\n result->c.numberOfNodesToFreeze = info->numberOfNodesToFreeze;\r\n DESC1_INITIALIZED(tid);\r\n return result;\r\n}\r\n\r\ntemplate \r\nabtree_ns::Node * abtree_ns::abtree::allocateNode(const int tid) {\r\n Node *newnode = recordmgr->template allocate >(tid);\r\n if (newnode == NULL) {\r\n COUTATOMICTID(\"ERROR: could not allocate node\"<init_node(tid, newnode);\r\n#ifdef __HANDLE_STATS\r\n GSTATS_APPEND(tid, node_allocated_addresses, ((long long) newnode)%(1<<12));\r\n#endif\r\n return newnode;\r\n}\r\n\r\n/**\r\n * Returns the value associated with key, or NULL if key is not present.\r\n */\r\ntemplate \r\nconst pair abtree_ns::abtree::find(const int tid, const K& key) {\r\n pair result;\r\n this->recordmgr->leaveQuiescentState(tid);\r\n Node * l = rqProvider->read_addr(tid, &entry->ptrs[0]);\r\n while (!l->isLeaf()) {\r\n int ix = l->getChildIndex(key, cmp);\r\n l = rqProvider->read_addr(tid, &l->ptrs[ix]);\r\n }\r\n int index = l->getKeyIndex(key, cmp);\r\n if (index < l->getKeyCount() && l->keys[index] == key) {\r\n result.first = l->ptrs[index]; // this is a value, not a pointer, so it cannot be modified by rqProvider->linearize_update_at_..., so we do not use read_addr\r\n result.second = true;\r\n } else {\r\n result.first = NO_VALUE;\r\n result.second = false;\r\n }\r\n this->recordmgr->enterQuiescentState(tid);\r\n return result;\r\n}\r\n\r\ntemplate \r\nbool abtree_ns::abtree::contains(const int tid, const K& key) {\r\n return find(tid, key).second;\r\n}\r\n\r\ntemplate\r\nint abtree_ns::abtree::rangeQuery(const int tid, const K& lo, const K& hi, K * const resultKeys, void ** const resultValues) {\r\n block> stack (NULL);\r\n recordmgr->leaveQuiescentState(tid);\r\n rqProvider->traversal_start(tid);\r\n\r\n // depth first traversal (of interesting subtrees)\r\n int size = 0;\r\n TRACE COUTATOMICTID(\"rangeQuery(lo=\"< * node = stack.pop();\r\n assert(node);\r\n \r\n // if leaf node, check if we should add its keys to the traversal\r\n if (node->isLeaf()) {\r\n rqProvider->traversal_try_add(tid, node, resultKeys, resultValues, &size, lo, hi);\r\n \r\n // else if internal node, explore its children\r\n } else {\r\n // find right-most sub-tree that could contain a key in [lo, hi]\r\n int nkeys = node->getKeyCount();\r\n int r = nkeys;\r\n while (r > 0 && cmp(hi, (const K&) node->keys[r-1])) --r; // subtree rooted at node->ptrs[r] contains only keys > hi\r\n\r\n // find left-most sub-tree that could contain a key in [lo, hi]\r\n int l = 0;\r\n while (l < nkeys && !cmp(lo, (const K&) node->keys[l])) ++l; // subtree rooted at node->ptrs[l] contains only keys < lo\r\n\r\n // perform DFS from left to right (so push onto stack from right to left)\r\n for (int i=r;i>=l; --i) stack.push(rqProvider->read_addr(tid, &node->ptrs[i]));\r\n\r\n// // simply explore EVERYTHING\r\n// for (int i=0;igetABDegree();++i) {\r\n// stack.push(rqProvider->read_addr(tid, &node->ptrs[i]));\r\n// }\r\n }\r\n }\r\n \r\n // success\r\n rqProvider->traversal_end(tid, resultKeys, resultValues, &size, lo, hi);\r\n recordmgr->enterQuiescentState(tid);\r\n return size;\r\n}\r\n\r\n\r\ntemplate \r\nvoid* abtree_ns::abtree::doInsert(const int tid, const K& key, void * const value, const bool replace) {\r\n wrapper_info _info;\r\n wrapper_info* info = &_info;\r\n while (true) {\r\n /**\r\n * search\r\n */\r\n this->recordmgr->leaveQuiescentState(tid);\r\n Node* gp = NULL;\r\n Node* p = entry;\r\n Node* l = rqProvider->read_addr(tid, &p->ptrs[0]);\r\n int ixToP = -1;\r\n int ixToL = 0;\r\n while (!l->isLeaf()) {\r\n ixToP = ixToL;\r\n ixToL = l->getChildIndex(key, cmp);\r\n gp = p;\r\n p = l;\r\n l = rqProvider->read_addr(tid, &l->ptrs[ixToL]);\r\n }\r\n\r\n /**\r\n * do the update\r\n */\r\n int keyIndex = l->getKeyIndex(key, cmp);\r\n if (keyIndex < l->getKeyCount() && l->keys[keyIndex] == key) {\r\n /**\r\n * if l already contains key, replace the existing value\r\n */\r\n void* const oldValue = l->ptrs[keyIndex]; // this is a value, not a pointer, so it cannot be modified by rqProvider->linearize_update_at_..., so we do not use read_addr\r\n if (!replace) {\r\n this->recordmgr->enterQuiescentState(tid);\r\n return oldValue;\r\n }\r\n \r\n // perform LLXs\r\n if (!llx(tid, p, NULL, 0, info->scxPtrs, info->nodes)\r\n || rqProvider->read_addr(tid, &p->ptrs[ixToL]) != l) {\r\n this->recordmgr->enterQuiescentState(tid);\r\n continue; // retry the search\r\n }\r\n info->nodes[1] = l;\r\n \r\n // create new node(s)\r\n Node* n = allocateNode(tid);\r\n arraycopy(l->keys, 0, n->keys, 0, l->getKeyCount());\r\n arraycopy(l->ptrs, 0, n->ptrs, 0, l->getABDegree()); // although we are copying l->ptrs, since l is a leaf, l->ptrs CANNOT contain modified by rqProvider->linearize_update_at_..., so we do not use arraycopy_ptrs.\r\n n->ptrs[keyIndex] = (Node*) value; // similarly, we don't use write_addr here\r\n n->leaf = true;\r\n n->marked = false;\r\n n->scxPtr = DUMMY;\r\n n->searchKey = l->searchKey;\r\n n->size = l->size;\r\n n->weight = true;\r\n \r\n // construct info record to pass to SCX\r\n info->numberOfNodes = 2;\r\n info->numberOfNodesAllocated = 1;\r\n info->numberOfNodesToFreeze = 1;\r\n info->field = &p->ptrs[ixToL];\r\n info->newNode = n;\r\n info->insertedNodes[0] = n;\r\n info->insertedNodes[1] = NULL;\r\n info->deletedNodes[0] = l;\r\n info->deletedNodes[1] = NULL;\r\n\r\n if (scx(tid, info)) {\r\n TRACE COUTATOMICTID(\"replace pair (\"<recordmgr->enterQuiescentState(tid);\r\n return oldValue;\r\n }\r\n TRACE COUTATOMICTID(\"replace pair (\"<recordmgr->enterQuiescentState(tid);\r\n this->recordmgr->deallocate(tid, n);\r\n\r\n } else {\r\n /**\r\n * if l does not contain key, we have to insert it\r\n */\r\n\r\n // perform LLXs\r\n if (!llx(tid, p, NULL, 0, info->scxPtrs, info->nodes) || rqProvider->read_addr(tid, &p->ptrs[ixToL]) != l) {\r\n this->recordmgr->enterQuiescentState(tid);\r\n continue; // retry the search\r\n }\r\n info->nodes[1] = l;\r\n \r\n if (l->getKeyCount() < b) {\r\n /**\r\n * Insert pair\r\n */\r\n \r\n // create new node(s)\r\n Node* n = allocateNode(tid);\r\n arraycopy(l->keys, 0, n->keys, 0, keyIndex);\r\n arraycopy(l->keys, keyIndex, n->keys, keyIndex+1, l->getKeyCount()-keyIndex);\r\n n->keys[keyIndex] = key;\r\n arraycopy(l->ptrs, 0, n->ptrs, 0, keyIndex); // although we are copying the ptrs array, since the source node is a leaf, ptrs CANNOT contain modified by rqProvider->linearize_update_at_..., so we do not use arraycopy_ptrs.\r\n arraycopy(l->ptrs, keyIndex, n->ptrs, keyIndex+1, l->getABDegree()-keyIndex);\r\n n->ptrs[keyIndex] = (Node*) value; // similarly, we don't use write_addr here\r\n n->leaf = l->leaf;\r\n n->marked = false;\r\n n->scxPtr = DUMMY;\r\n n->searchKey = l->searchKey;\r\n n->size = l->size+1;\r\n n->weight = l->weight;\r\n\r\n // construct info record to pass to SCX\r\n info->numberOfNodes = 2;\r\n info->numberOfNodesAllocated = 1;\r\n info->numberOfNodesToFreeze = 1;\r\n info->field = &p->ptrs[ixToL];\r\n info->newNode = n;\r\n info->insertedNodes[0] = n;\r\n info->insertedNodes[1] = NULL;\r\n info->deletedNodes[0] = l;\r\n info->deletedNodes[1] = NULL;\r\n \r\n if (scx(tid, info)) {\r\n TRACE COUTATOMICTID(\"insert pair (\"<recordmgr->enterQuiescentState(tid);\r\n return NO_VALUE;\r\n }\r\n TRACE COUTATOMICTID(\"insert pair (\"<recordmgr->enterQuiescentState(tid);\r\n this->recordmgr->deallocate(tid, n);\r\n \r\n } else { // assert: l->getKeyCount() == DEGREE == b)\r\n /**\r\n * Overflow\r\n */\r\n \r\n // first, we create a pair of large arrays\r\n // containing too many keys and pointers to fit in a single node\r\n K keys[DEGREE+1];\r\n Node* ptrs[DEGREE+1];\r\n arraycopy(l->keys, 0, keys, 0, keyIndex);\r\n arraycopy(l->keys, keyIndex, keys, keyIndex+1, l->getKeyCount()-keyIndex);\r\n keys[keyIndex] = key;\r\n arraycopy(l->ptrs, 0, ptrs, 0, keyIndex); // although we are copying the ptrs array, since the source node is a leaf, ptrs CANNOT contain modified by rqProvider->linearize_update_at_..., so we do not use arraycopy_ptrs.\r\n arraycopy(l->ptrs, keyIndex, ptrs, keyIndex+1, l->getABDegree()-keyIndex);\r\n ptrs[keyIndex] = (Node*) value;\r\n\r\n // create new node(s):\r\n // since the new arrays are too big to fit in a single node,\r\n // we replace l by a new subtree containing three new nodes:\r\n // a parent, and two leaves;\r\n // the array contents are then split between the two new leaves\r\n\r\n const int size1 = (DEGREE+1)/2;\r\n Node* left = allocateNode(tid);\r\n arraycopy(keys, 0, left->keys, 0, size1);\r\n arraycopy(ptrs, 0, left->ptrs, 0, size1); // although we are copying the ptrs array, since the node is a leaf, ptrs CANNOT contain modified by rqProvider->linearize_update_at_..., so we do not use arraycopy_ptrs.\r\n left->leaf = true;\r\n left->marked = false;\r\n left->scxPtr = DUMMY;\r\n left->searchKey = keys[0];\r\n left->size = size1;\r\n left->weight = true;\r\n\r\n const int size2 = (DEGREE+1) - size1;\r\n Node* right = allocateNode(tid);\r\n arraycopy(keys, size1, right->keys, 0, size2);\r\n arraycopy(ptrs, size1, right->ptrs, 0, size2); // although we are copying the ptrs array, since the node is a leaf, ptrs CANNOT contain modified by rqProvider->linearize_update_at_..., so we do not use arraycopy_ptrs.\r\n right->leaf = true;\r\n right->marked = false;\r\n right->scxPtr = DUMMY;\r\n right->searchKey = keys[size1];\r\n right->size = size2;\r\n right->weight = true;\r\n \r\n Node* n = allocateNode(tid);\r\n n->keys[0] = keys[size1];\r\n rqProvider->write_addr(tid, &n->ptrs[0], left);\r\n rqProvider->write_addr(tid, &n->ptrs[1], right);\r\n n->leaf = false;\r\n n->marked = false;\r\n n->scxPtr = DUMMY;\r\n n->searchKey = keys[size1];\r\n n->size = 2;\r\n n->weight = p == entry;\r\n \r\n // note: weight of new internal node n will be zero,\r\n // unless it is the root; this is because we test\r\n // p == entry, above; in doing this, we are actually\r\n // performing Root-Zero at the same time as this Overflow\r\n // if n will become the root (of the B-slack tree)\r\n \r\n // construct info record to pass to SCX\r\n info->numberOfNodes = 2;\r\n info->numberOfNodesAllocated = 3;\r\n info->numberOfNodesToFreeze = 1;\r\n info->field = &p->ptrs[ixToL];\r\n info->newNode = n;\r\n info->insertedNodes[0] = n;\r\n info->insertedNodes[1] = left;\r\n info->insertedNodes[2] = right;\r\n info->insertedNodes[3] = NULL;\r\n info->deletedNodes[0] = l;\r\n info->deletedNodes[1] = NULL;\r\n\r\n if (scx(tid, info)) {\r\n TRACE COUTATOMICTID(\"insert overflow (\"<recordmgr->enterQuiescentState(tid);\r\n return NO_VALUE;\r\n }\r\n TRACE COUTATOMICTID(\"insert overflow (\"<recordmgr->enterQuiescentState(tid);\r\n this->recordmgr->deallocate(tid, n);\r\n this->recordmgr->deallocate(tid, left);\r\n this->recordmgr->deallocate(tid, right);\r\n }\r\n }\r\n }\r\n}\r\n\r\ntemplate \r\nconst pair abtree_ns::abtree::erase(const int tid, const K& key) {\r\n wrapper_info _info;\r\n wrapper_info* info = &_info;\r\n while (true) {\r\n /**\r\n * search\r\n */\r\n this->recordmgr->leaveQuiescentState(tid);\r\n Node* gp = NULL;\r\n Node* p = entry;\r\n Node* l = rqProvider->read_addr(tid, &p->ptrs[0]);\r\n int ixToP = -1;\r\n int ixToL = 0;\r\n while (!l->isLeaf()) {\r\n ixToP = ixToL;\r\n ixToL = l->getChildIndex(key, cmp);\r\n gp = p;\r\n p = l;\r\n l = rqProvider->read_addr(tid, &l->ptrs[ixToL]);\r\n }\r\n\r\n /**\r\n * do the update\r\n */\r\n const int keyIndex = l->getKeyIndex(key, cmp);\r\n if (keyIndex == l->getKeyCount() || l->keys[keyIndex] != key) {\r\n /**\r\n * if l does not contain key, we are done.\r\n */\r\n this->recordmgr->enterQuiescentState(tid);\r\n return pair(NO_VALUE,false);\r\n } else {\r\n /**\r\n * if l contains key, replace l by a new copy that does not contain key.\r\n */\r\n\r\n // perform LLXs\r\n if (!llx(tid, p, NULL, 0, info->scxPtrs, info->nodes) || rqProvider->read_addr(tid, &p->ptrs[ixToL]) != l) {\r\n this->recordmgr->enterQuiescentState(tid);\r\n continue; // retry the search\r\n }\r\n info->nodes[1] = l;\r\n // create new node(s)\r\n Node* n = allocateNode(tid);\r\n //printf(\"keyIndex=%d getABDegree-keyIndex=%d\\n\", keyIndex, l->getABDegree()-keyIndex);\r\n arraycopy(l->keys, 0, n->keys, 0, keyIndex);\r\n arraycopy(l->keys, keyIndex+1, n->keys, keyIndex, l->getKeyCount()-(keyIndex+1));\r\n arraycopy(l->ptrs, 0, n->ptrs, 0, keyIndex); // although we are copying the ptrs array, since the node is a leaf, ptrs CANNOT contain modified by rqProvider->linearize_update_at_..., so we do not use arraycopy_ptrs.\r\n arraycopy(l->ptrs, keyIndex+1, n->ptrs, keyIndex, l->getABDegree()-(keyIndex+1));\r\n n->leaf = true;\r\n n->marked = false;\r\n n->scxPtr = DUMMY;\r\n n->searchKey = l->keys[0]; // NOTE: WE MIGHT BE DELETING l->keys[0], IN WHICH CASE newL IS EMPTY. HOWEVER, newL CAN STILL BE LOCATED BY SEARCHING FOR l->keys[0], SO WE USE THAT AS THE searchKey FOR newL.\r\n n->size = l->size-1;\r\n n->weight = true;\r\n\r\n // construct info record to pass to SCX\r\n info->numberOfNodes = 2;\r\n info->numberOfNodesAllocated = 1;\r\n info->numberOfNodesToFreeze = 1;\r\n info->field = &p->ptrs[ixToL];\r\n info->newNode = n;\r\n info->insertedNodes[0] = n;\r\n info->insertedNodes[1] = NULL;\r\n info->deletedNodes[0] = l;\r\n info->deletedNodes[1] = NULL;\r\n\r\n void* oldValue = l->ptrs[keyIndex]; // since the node is a leaf, ptrs is not modified by any call to rqProvider->linearize_update_at_..., so we do not need to use read_addr to access it\r\n if (scx(tid, info)) {\r\n TRACE COUTATOMICTID(\"delete pair (\"<recordmgr->enterQuiescentState(tid);\r\n return pair(oldValue, true);\r\n }\r\n TRACE COUTATOMICTID(\"delete pair (\"<recordmgr->enterQuiescentState(tid);\r\n this->recordmgr->deallocate(tid, n);\r\n }\r\n }\r\n}\r\n\r\n/**\r\n * \r\n * \r\n * IMPLEMENTATION OF REBALANCING\r\n * \r\n * \r\n */\r\n\r\ntemplate \r\nbool abtree_ns::abtree::fixWeightViolation(const int tid, Node* viol) {\r\n if (viol->weight) return false;\r\n\r\n // assert: viol is internal (because leaves always have weight = 1)\r\n // assert: viol is not entry or root (because both always have weight = 1)\r\n\r\n // do an optimistic check to see if viol was already removed from the tree\r\n if (llx(tid, viol, NULL) == FINALIZED) {\r\n // recall that nodes are finalized precisely when\r\n // they are removed from the tree\r\n // we hand off responsibility for any violations at viol to the\r\n // process that removed it.\r\n return false;\r\n }\r\n\r\n wrapper_info _info;\r\n wrapper_info* info = &_info;\r\n\r\n // try to locate viol, and fix any weight violation at viol\r\n while (true) {\r\n\r\n const K k = viol->searchKey;\r\n Node* gp = NULL;\r\n Node* p = entry;\r\n Node* l = rqProvider->read_addr(tid, &p->ptrs[0]);\r\n int ixToP = -1;\r\n int ixToL = 0;\r\n while (!l->isLeaf() && l != viol) {\r\n ixToP = ixToL;\r\n ixToL = l->getChildIndex(k, cmp);\r\n gp = p;\r\n p = l;\r\n l = rqProvider->read_addr(tid, &l->ptrs[ixToL]);\r\n }\r\n\r\n if (l != viol) {\r\n // l was replaced by another update.\r\n // we hand over responsibility for viol to that update.\r\n return false;\r\n }\r\n\r\n // we cannot apply this update if p has a weight violation\r\n // so, we check if this is the case, and, if so, try to fix it\r\n if (!p->weight) {\r\n fixWeightViolation(tid, p);\r\n continue;\r\n }\r\n \r\n // perform LLXs\r\n if (!llx(tid, gp, NULL, 0, info->scxPtrs, info->nodes) || rqProvider->read_addr(tid, &gp->ptrs[ixToP]) != p) continue; // retry the search\r\n if (!llx(tid, p, NULL, 1, info->scxPtrs, info->nodes) || rqProvider->read_addr(tid, &p->ptrs[ixToL]) != l) continue; // retry the search\r\n if (!llx(tid, l, NULL, 2, info->scxPtrs, info->nodes)) continue; // retry the search\r\n\r\n const int c = p->getABDegree() + l->getABDegree();\r\n const int size = c-1;\r\n\r\n if (size <= b) {\r\n /**\r\n * Absorb\r\n */\r\n\r\n // create new node(s)\r\n // the new arrays are small enough to fit in a single node,\r\n // so we replace p by a new internal node.\r\n Node* n = allocateNode(tid);\r\n arraycopy_ptrs(p->ptrs, 0, n->ptrs, 0, ixToL); // p and l are both internal, so we use arraycopy_ptrs\r\n arraycopy_ptrs(l->ptrs, 0, n->ptrs, ixToL, l->getABDegree());\r\n arraycopy_ptrs(p->ptrs, ixToL+1, n->ptrs, ixToL+l->getABDegree(), p->getABDegree()-(ixToL+1));\r\n arraycopy(p->keys, 0, n->keys, 0, ixToL);\r\n arraycopy(l->keys, 0, n->keys, ixToL, l->getKeyCount());\r\n arraycopy(p->keys, ixToL, n->keys, ixToL+l->getKeyCount(), p->getKeyCount()-ixToL);\r\n n->leaf = false; assert(!l->isLeaf());\r\n n->marked = false;\r\n n->scxPtr = DUMMY;\r\n n->searchKey = n->keys[0];\r\n n->size = size;\r\n n->weight = true;\r\n \r\n // construct info record to pass to SCX\r\n info->numberOfNodes = 3;\r\n info->numberOfNodesAllocated = 1;\r\n info->numberOfNodesToFreeze = 3;\r\n info->field = &gp->ptrs[ixToP];\r\n info->newNode = n;\r\n// info->insertedNodes[0] = info->deletedNodes[0] = NULL;\r\n info->insertedNodes[0] = n;\r\n info->insertedNodes[1] = NULL;\r\n info->deletedNodes[0] = p;\r\n info->deletedNodes[1] = l;\r\n info->deletedNodes[2] = NULL;\r\n \r\n if (scx(tid, info)) {\r\n TRACE COUTATOMICTID(\"absorb: SCX succeeded\"< eliminated\r\n // slack at pi(u) -> eliminated or slack at n\r\n // weight at u -> eliminated\r\n // no degree at u\r\n // slack at u -> slack at n\r\n\r\n /**\r\n * Compress may be needed at the new internal node we created\r\n * (since we move grandchildren from two parents together).\r\n */\r\n fixDegreeViolation(tid, n);\r\n return true;\r\n }\r\n TRACE COUTATOMICTID(\"absorb: SCX FAILED\"<recordmgr->deallocate(tid, n);\r\n\r\n } else {\r\n /**\r\n * Split\r\n */\r\n\r\n // merge keys of p and l into one big array (and similarly for children)\r\n // (we essentially replace the pointer to l with the contents of l)\r\n K keys[2*DEGREE];\r\n Node* ptrs[2*DEGREE];\r\n arraycopy_ptrs(p->ptrs, 0, ptrs, 0, ixToL); // p and l are both internal, so we use arraycopy_ptrs\r\n arraycopy_ptrs(l->ptrs, 0, ptrs, ixToL, l->getABDegree());\r\n arraycopy_ptrs(p->ptrs, ixToL+1, ptrs, ixToL+l->getABDegree(), p->getABDegree()-(ixToL+1));\r\n arraycopy(p->keys, 0, keys, 0, ixToL);\r\n arraycopy(l->keys, 0, keys, ixToL, l->getKeyCount());\r\n arraycopy(p->keys, ixToL, keys, ixToL+l->getKeyCount(), p->getKeyCount()-ixToL);\r\n\r\n // the new arrays are too big to fit in a single node,\r\n // so we replace p by a new internal node and two new children.\r\n //\r\n // we take the big merged array and split it into two arrays,\r\n // which are used to create two new children u and v.\r\n // we then create a new internal node (whose weight will be zero\r\n // if it is not the root), with u and v as its children.\r\n \r\n // create new node(s)\r\n const int size1 = size / 2;\r\n Node* left = allocateNode(tid);\r\n arraycopy(keys, 0, left->keys, 0, size1-1);\r\n arraycopy_ptrs(ptrs, 0, left->ptrs, 0, size1);\r\n left->leaf = false; assert(!l->isLeaf());\r\n left->marked = false;\r\n left->scxPtr = DUMMY;\r\n left->searchKey = keys[0];\r\n left->size = size1;\r\n left->weight = true;\r\n\r\n const int size2 = size - size1;\r\n Node* right = allocateNode(tid);\r\n arraycopy(keys, size1, right->keys, 0, size2-1);\r\n arraycopy_ptrs(ptrs, size1, right->ptrs, 0, size2);\r\n right->leaf = false;\r\n right->marked = false;\r\n right->scxPtr = DUMMY;\r\n right->searchKey = keys[size1];\r\n right->size = size2;\r\n right->weight = true;\r\n\r\n Node* n = allocateNode(tid);\r\n n->keys[0] = keys[size1-1];\r\n rqProvider->write_addr(tid, &n->ptrs[0], left);\r\n rqProvider->write_addr(tid, &n->ptrs[1], right);\r\n n->leaf = false;\r\n n->marked = false;\r\n n->scxPtr = DUMMY;\r\n n->searchKey = keys[size1-1]; // note: should be the same as n->keys[0]\r\n n->size = 2;\r\n n->weight = (gp == entry);\r\n\r\n // note: weight of new internal node n will be zero,\r\n // unless it is the root; this is because we test\r\n // gp == entry, above; in doing this, we are actually\r\n // performing Root-Zero at the same time as this Overflow\r\n // if n will become the root (of the B-slack tree)\r\n\r\n // construct info record to pass to SCX\r\n info->numberOfNodes = 3;\r\n info->numberOfNodesAllocated = 3;\r\n info->numberOfNodesToFreeze = 3;\r\n info->field = &gp->ptrs[ixToP];\r\n info->newNode = n;\r\n// info->insertedNodes[0] = info->deletedNodes[0] = NULL;\r\n info->insertedNodes[0] = n;\r\n info->insertedNodes[1] = left;\r\n info->insertedNodes[2] = right;\r\n info->insertedNodes[3] = NULL;\r\n info->deletedNodes[0] = p;\r\n info->deletedNodes[1] = l;\r\n info->deletedNodes[2] = NULL;\r\n\r\n if (scx(tid, info)) {\r\n TRACE COUTATOMICTID(\"split: SCX succeeded\"<recordmgr->deallocate(tid, n);\r\n this->recordmgr->deallocate(tid, left);\r\n this->recordmgr->deallocate(tid, right);\r\n }\r\n }\r\n}\r\n\r\ntemplate \r\nbool abtree_ns::abtree::fixDegreeViolation(const int tid, Node* viol) {\r\n if (viol->getABDegree() >= a || viol == entry || viol == rqProvider->read_addr(tid, &entry->ptrs[0])) {\r\n return false; // no degree violation at viol\r\n }\r\n \r\n // do an optimistic check to see if viol was already removed from the tree\r\n if (llx(tid, viol, NULL) == FINALIZED) {\r\n // recall that nodes are finalized precisely when\r\n // they are removed from the tree.\r\n // we hand off responsibility for any violations at viol to the\r\n // process that removed it.\r\n return false;\r\n }\r\n\r\n wrapper_info _info;\r\n wrapper_info* info = &_info;\r\n\r\n // we search for viol and try to fix any violation we find there\r\n // this entails performing AbsorbSibling or Distribute.\r\n while (true) {\r\n /**\r\n * search for viol\r\n */\r\n const K k = viol->searchKey;\r\n Node* gp = NULL;\r\n Node* p = entry;\r\n Node* l = rqProvider->read_addr(tid, &p->ptrs[0]);\r\n int ixToP = -1;\r\n int ixToL = 0;\r\n while (!l->isLeaf() && l != viol) {\r\n ixToP = ixToL;\r\n ixToL = l->getChildIndex(k, cmp);\r\n gp = p;\r\n p = l;\r\n l = rqProvider->read_addr(tid, &l->ptrs[ixToL]);\r\n }\r\n\r\n if (l != viol) {\r\n // l was replaced by another update.\r\n // we hand over responsibility for viol to that update.\r\n return false;\r\n }\r\n \r\n // assert: gp != NULL (because if AbsorbSibling or Distribute can be applied, then p is not the root)\r\n \r\n // perform LLXs\r\n if (!llx(tid, gp, NULL, 0, info->scxPtrs, info->nodes)\r\n || rqProvider->read_addr(tid, &gp->ptrs[ixToP]) != p) continue; // retry the search\r\n if (!llx(tid, p, NULL, 1, info->scxPtrs, info->nodes) \r\n || rqProvider->read_addr(tid, &p->ptrs[ixToL]) != l) continue; // retry the search\r\n\r\n int ixToS = (ixToL > 0 ? ixToL-1 : 1);\r\n Node* s = rqProvider->read_addr(tid, &p->ptrs[ixToS]);\r\n \r\n // we can only apply AbsorbSibling or Distribute if there are no\r\n // weight violations at p, l or s.\r\n // so, we first check for any weight violations,\r\n // and fix any that we see.\r\n bool foundWeightViolation = false;\r\n if (!p->weight) {\r\n foundWeightViolation = true;\r\n fixWeightViolation(tid, p);\r\n }\r\n if (!l->weight) {\r\n foundWeightViolation = true;\r\n fixWeightViolation(tid, l);\r\n }\r\n if (!s->weight) {\r\n foundWeightViolation = true;\r\n fixWeightViolation(tid, s);\r\n }\r\n // if we see any weight violations, then either we fixed one,\r\n // removing one of these nodes from the tree,\r\n // or one of the nodes has been removed from the tree by another\r\n // rebalancing step, so we retry the search for viol\r\n if (foundWeightViolation) continue;\r\n\r\n // assert: there are no weight violations at p, l or s\r\n // assert: l and s are either both leaves or both internal nodes\r\n // (because there are no weight violations at these nodes)\r\n \r\n // also note that p->size >= a >= 2\r\n \r\n Node* left;\r\n Node* right;\r\n int leftindex;\r\n int rightindex;\r\n\r\n if (ixToL < ixToS) {\r\n if (!llx(tid, l, NULL, 2, info->scxPtrs, info->nodes)) continue; // retry the search\r\n if (!llx(tid, s, NULL, 3, info->scxPtrs, info->nodes)) continue; // retry the search\r\n left = l;\r\n right = s;\r\n leftindex = ixToL;\r\n rightindex = ixToS;\r\n } else {\r\n if (!llx(tid, s, NULL, 2, info->scxPtrs, info->nodes)) continue; // retry the search\r\n if (!llx(tid, l, NULL, 3, info->scxPtrs, info->nodes)) continue; // retry the search\r\n left = s;\r\n right = l;\r\n leftindex = ixToS;\r\n rightindex = ixToL;\r\n }\r\n \r\n int sz = left->getABDegree() + right->getABDegree();\r\n assert(left->weight && right->weight);\r\n \r\n if (sz < 2*a) {\r\n /**\r\n * AbsorbSibling\r\n */\r\n \r\n // create new node(s))\r\n Node* newl = allocateNode(tid);\r\n int k1=0, k2=0;\r\n for (int i=0;igetKeyCount();++i) {\r\n newl->keys[k1++] = left->keys[i];\r\n }\r\n for (int i=0;igetABDegree();++i) {\r\n if (left->isLeaf()) {\r\n newl->ptrs[k2++] = left->ptrs[i];\r\n } else {\r\n //assert(left->getKeyCount() != left->getABDegree());\r\n rqProvider->write_addr(tid, &newl->ptrs[k2++], rqProvider->read_addr(tid, &left->ptrs[i]));\r\n }\r\n }\r\n if (!left->isLeaf()) newl->keys[k1++] = p->keys[leftindex];\r\n for (int i=0;igetKeyCount();++i) {\r\n newl->keys[k1++] = right->keys[i];\r\n }\r\n for (int i=0;igetABDegree();++i) {\r\n if (right->isLeaf()) {\r\n newl->ptrs[k2++] = right->ptrs[i];\r\n } else {\r\n rqProvider->write_addr(tid, &newl->ptrs[k2++], rqProvider->read_addr(tid, &right->ptrs[i]));\r\n }\r\n }\r\n newl->leaf = left->isLeaf();\r\n newl->marked = false;\r\n newl->scxPtr = DUMMY;\r\n newl->searchKey = l->searchKey;\r\n newl->size = l->getABDegree() + s->getABDegree();\r\n newl->weight = true; assert(left->weight && right->weight && p->weight);\r\n \r\n // now, we atomically replace p and its children with the new nodes.\r\n // if appropriate, we perform RootAbsorb at the same time.\r\n if (gp == entry && p->getABDegree() == 2) {\r\n \r\n // construct info record to pass to SCX\r\n info->numberOfNodes = 4; // gp + p + l + s\r\n info->numberOfNodesAllocated = 1; // newl\r\n info->numberOfNodesToFreeze = 4; // gp + p + l + s\r\n info->field = &gp->ptrs[ixToP];\r\n info->newNode = newl;\r\n info->insertedNodes[0] = newl;\r\n info->insertedNodes[1] = NULL;\r\n info->deletedNodes[0] = p;\r\n info->deletedNodes[1] = l;\r\n info->deletedNodes[2] = s;\r\n info->deletedNodes[3] = NULL;\r\n \r\n if (scx(tid, info)) {\r\n TRACE COUTATOMICTID(\"absorbsibling AND rootabsorb: SCX succeeded\"<recordmgr->deallocate(tid, newl);\r\n \r\n } else {\r\n assert(gp != entry || p->getABDegree() > 2);\r\n \r\n // create n from p by:\r\n // 1. skipping the key for leftindex and child pointer for ixToS\r\n // 2. replacing l with newl\r\n Node* n = allocateNode(tid);\r\n for (int i=0;ikeys[i] = p->keys[i];\r\n }\r\n for (int i=0;iwrite_addr(tid, &n->ptrs[i], rqProvider->read_addr(tid, &p->ptrs[i])); // n and p are internal, so their ptrs arrays might have entries that are being modified by rqProvider->linearize_update_at_..., so we use read_addr and write_addr\r\n }\r\n for (int i=leftindex+1;igetKeyCount();++i) {\r\n n->keys[i-1] = p->keys[i];\r\n }\r\n for (int i=ixToL+1;igetABDegree();++i) {\r\n rqProvider->write_addr(tid, &n->ptrs[i-1], rqProvider->read_addr(tid, &p->ptrs[i])); // n and p are internal, so their ptrs arrays might have entries that are being modified by rqProvider->linearize_update_at_..., so we use read_addr and write_addr\r\n }\r\n // replace l with newl\r\n rqProvider->write_addr(tid, &n->ptrs[ixToL - (ixToL > ixToS)], newl);\r\n n->leaf = false;\r\n n->marked = false;\r\n n->scxPtr = DUMMY;\r\n n->searchKey = p->searchKey;\r\n n->size = p->getABDegree()-1;\r\n n->weight = true;\r\n\r\n // construct info record to pass to SCX\r\n info->numberOfNodes = 4; // gp + p + l + s\r\n info->numberOfNodesAllocated = 2; // n + newl\r\n info->numberOfNodesToFreeze = 4; // gp + p + l + s\r\n info->field = &gp->ptrs[ixToP];\r\n info->newNode = n;\r\n info->insertedNodes[0] = n;\r\n info->insertedNodes[1] = newl;\r\n info->insertedNodes[2] = NULL;\r\n info->deletedNodes[0] = p;\r\n info->deletedNodes[1] = l;\r\n info->deletedNodes[2] = s;\r\n info->deletedNodes[3] = NULL;\r\n \r\n if (scx(tid, info)) {\r\n TRACE COUTATOMICTID(\"absorbsibling: SCX succeeded\"<recordmgr->deallocate(tid, newl);\r\n this->recordmgr->deallocate(tid, n);\r\n }\r\n \r\n } else {\r\n /**\r\n * Distribute\r\n */\r\n \r\n int leftsz = sz/2;\r\n int rightsz = sz-leftsz;\r\n \r\n // create new node(s))\r\n Node* n = allocateNode(tid);\r\n Node* newleft = allocateNode(tid);\r\n Node* newright = allocateNode(tid);\r\n \r\n // combine the contents of l and s (and one key from p if l and s are internal)\r\n K keys[2*DEGREE];\r\n Node* ptrs[2*DEGREE];\r\n int k1=0, k2=0;\r\n for (int i=0;igetKeyCount();++i) {\r\n keys[k1++] = left->keys[i];\r\n }\r\n for (int i=0;igetABDegree();++i) {\r\n if (left->isLeaf()) {\r\n ptrs[k2++] = left->ptrs[i];\r\n } else {\r\n ptrs[k2++] = rqProvider->read_addr(tid, &left->ptrs[i]);\r\n }\r\n }\r\n if (!left->isLeaf()) keys[k1++] = p->keys[leftindex];\r\n for (int i=0;igetKeyCount();++i) {\r\n keys[k1++] = right->keys[i];\r\n }\r\n for (int i=0;igetABDegree();++i) {\r\n if (right->isLeaf()) {\r\n ptrs[k2++] = right->ptrs[i];\r\n } else {\r\n ptrs[k2++] = rqProvider->read_addr(tid, &right->ptrs[i]);\r\n }\r\n }\r\n \r\n // distribute contents between newleft and newright\r\n k1=0;\r\n k2=0;\r\n for (int i=0;iisLeaf();++i) {\r\n newleft->keys[i] = keys[k1++];\r\n }\r\n for (int i=0;iisLeaf()) {\r\n newleft->ptrs[i] = ptrs[k2++];\r\n } else {\r\n rqProvider->write_addr(tid, &newleft->ptrs[i], ptrs[k2++]);\r\n }\r\n }\r\n newleft->leaf = left->isLeaf();\r\n newleft->marked = false;\r\n newleft->scxPtr = DUMMY;\r\n newleft->searchKey = newleft->keys[0];\r\n newleft->size = leftsz;\r\n newleft->weight = true;\r\n \r\n // reserve one key for the parent (to go between newleft and newright)\r\n K keyp = keys[k1];\r\n if (!left->isLeaf()) ++k1;\r\n for (int i=0;iisLeaf();++i) {\r\n newright->keys[i] = keys[k1++];\r\n }\r\n for (int i=0;iisLeaf()) {\r\n newright->ptrs[i] = ptrs[k2++];\r\n } else {\r\n rqProvider->write_addr(tid, &newright->ptrs[i], ptrs[k2++]);\r\n }\r\n }\r\n newright->leaf = right->isLeaf();\r\n newright->marked = false;\r\n newright->scxPtr = DUMMY;\r\n newright->searchKey = newright->keys[0];\r\n newright->size = rightsz;\r\n newright->weight = true;\r\n \r\n // create n from p by replacing left with newleft and right with newright,\r\n // and replacing one key (between these two pointers)\r\n for (int i=0;igetKeyCount();++i) {\r\n n->keys[i] = p->keys[i];\r\n }\r\n for (int i=0;igetABDegree();++i) {\r\n rqProvider->write_addr(tid, &n->ptrs[i], rqProvider->read_addr(tid, &p->ptrs[i])); // n and p are internal, so their ptrs arrays might have entries that are being modified by rqProvider->linearize_update_at_..., so we use read_addr and write_addr\r\n }\r\n n->keys[leftindex] = keyp;\r\n rqProvider->write_addr(tid, &n->ptrs[leftindex], newleft);\r\n rqProvider->write_addr(tid, &n->ptrs[rightindex], newright);\r\n n->leaf = false;\r\n n->marked = false;\r\n n->scxPtr = DUMMY;\r\n n->searchKey = p->searchKey;\r\n n->size = p->size;\r\n n->weight = true;\r\n \r\n // construct info record to pass to SCX\r\n info->numberOfNodes = 4; // gp + p + l + s\r\n info->numberOfNodesAllocated = 3; // n + newleft + newright\r\n info->numberOfNodesToFreeze = 4; // gp + p + l + s\r\n info->field = &gp->ptrs[ixToP];\r\n info->newNode = n;\r\n info->insertedNodes[0] = n;\r\n info->insertedNodes[1] = newleft;\r\n info->insertedNodes[2] = newright;\r\n info->insertedNodes[3] = NULL;\r\n info->deletedNodes[0] = p;\r\n info->deletedNodes[1] = l;\r\n info->deletedNodes[2] = s;\r\n info->deletedNodes[3] = NULL;\r\n \r\n if (scx(tid, info)) {\r\n TRACE COUTATOMICTID(\"distribute: SCX succeeded\"<recordmgr->deallocate(tid, n);\r\n this->recordmgr->deallocate(tid, newleft);\r\n this->recordmgr->deallocate(tid, newright);\r\n }\r\n }\r\n}\r\n\r\n/**\r\n * \r\n * IMPLEMENTATION OF LLX AND SCX\r\n * \r\n * \r\n */\r\n\r\ntemplate \r\nbool abtree_ns::abtree::llx(const int tid, Node* r, Node ** snapshot, const int i, SCXRecord ** ops, Node ** nodes) {\r\n SCXRecord* result = llx(tid, r, snapshot);\r\n if (result == FAILED || result == FINALIZED) return false;\r\n ops[i] = result;\r\n nodes[i] = r;\r\n return true;\r\n}\r\n\r\ntemplate \r\nabtree_ns::SCXRecord* abtree_ns::abtree::llx(const int tid, Node* r, Node ** snapshot) {\r\n const bool marked = r->marked;\r\n SOFTWARE_BARRIER;\r\n tagptr_t tagptr = (tagptr_t) r->scxPtr;\r\n \r\n // read mutable state field of descriptor\r\n bool succ;\r\n TRACE COUTATOMICTID(\"tagged ptr seq=\"<c.mutables)<c.mutables, tagptr, MUTABLES1_MASK_STATE, MUTABLES1_OFFSET_STATE);\r\n if (!succ) state = SCXRecord::STATE_COMMITTED;\r\n TRACE { mutables_t debugmutables = TAGPTR1_UNPACK_PTR(tagptr)->c.mutables; COUTATOMICTID(\"llx scxrecord succ=\"<::STATE_ABORTED || ((state == SCXRecord::STATE_COMMITTED) && !r->marked)) {\r\n // read snapshot fields\r\n if (snapshot != NULL) {\r\n if (r->isLeaf()) {\r\n arraycopy(r->ptrs, 0, snapshot, 0, r->getABDegree());\r\n } else {\r\n arraycopy_ptrs(r->ptrs, 0, snapshot, 0, r->getABDegree());\r\n }\r\n }\r\n if ((tagptr_t) r->scxPtr == tagptr) return (SCXRecord *) tagptr; // we have a snapshot\r\n }\r\n\r\n if (state == SCXRecord::STATE_INPROGRESS) {\r\n helpOther(tid, tagptr);\r\n }\r\n return (marked ? FINALIZED : FAILED);\r\n}\r\n\r\ntemplate\r\nbool abtree_ns::abtree::scx(const int tid, wrapper_info * info) {\r\n const int init_state = SCXRecord::STATE_INPROGRESS;\r\n SCXRecord * newdesc = createSCXRecord(tid, info);\r\n tagptr_t tagptr = TAGPTR1_NEW(tid, newdesc->c.mutables);\r\n info->state = help(tid, tagptr, newdesc, false);\r\n return info->state & SCXRecord::STATE_COMMITTED;\r\n}\r\n\r\n// returns true if we executed help, and false otherwise\r\ntemplate\r\nvoid abtree_ns::abtree::helpOther(const int tid, tagptr_t tagptr) {\r\n if ((void*) tagptr == DUMMY) {\r\n return; // deal with the dummy descriptor\r\n }\r\n SCXRecord snap;\r\n if (DESC1_SNAPSHOT(&snap, tagptr, SCXRecord::size)) {\r\n help(tid, tagptr, &snap, true);\r\n }\r\n}\r\n\r\ntemplate\r\nint abtree_ns::abtree::help(const int tid, const tagptr_t tagptr, SCXRecord const * const snap, const bool helpingOther) {\r\n#ifdef NO_HELPING\r\n int IGNORED_RETURN_VALUE = -1;\r\n if (helpingOther) return IGNORED_RETURN_VALUE;\r\n#endif\r\n// TRACE COUTATOMICTID(\"help \"<c.numberOfNodes<<\" numToFreeze=\"<c.numberOfNodesToFreeze< *ptr = TAGPTR1_UNPACK_PTR(tagptr);\r\n //if (helpingOther) { eassert(UNPACK1_SEQ(snap->c.mutables), UNPACK1_SEQ(tagptr)); /*assert(UNPACK1_SEQ(snap->c.mutables) == UNPACK1_SEQ(tagptr));*/ }\r\n // freeze sub-tree\r\n for (int i=helpingOther; ic.numberOfNodesToFreeze; ++i) {\r\n if (snap->c.nodes[i]->isLeaf()) {\r\n TRACE COUTATOMICTID((helpingOther?\" \":\"\")<<\"help \"<<\"nodes[\"<c.nodes[i]))<<\" is a leaf\\n\");\r\n assert(i > 0); // nodes[0] cannot be a leaf...\r\n continue; // do not freeze leaves\r\n }\r\n \r\n bool successfulCAS = __sync_bool_compare_and_swap(&snap->c.nodes[i]->scxPtr, snap->c.scxPtrsSeen[i], tagptr);\r\n SCXRecord *exp = snap->c.nodes[i]->scxPtr;\r\n// TRACE if (successfulCAS) COUTATOMICTID((helpingOther?\" \":\"\")<<\"help froze nodes[\"<c.nodes[i])<<\" with tagptr=\"<c.nodes[i]->scxPtr)<c.mutables, tagptr, MUTABLES1_MASK_ALLFROZEN, MUTABLES1_OFFSET_ALLFROZEN);\r\n if (!succ) return SCXRecord::STATE_ABORTED;\r\n \r\n if (allFrozen) {\r\n TRACE COUTATOMICTID((helpingOther?\" \":\"\")<<\"help return state \"<::STATE_COMMITTED<<\" after failed freezing cas on nodes[\"<::STATE_COMMITTED;\r\n } else {\r\n const int newState = SCXRecord::STATE_ABORTED;\r\n TRACE COUTATOMICTID((helpingOther?\" \":\"\")<<\"help return state \"<c.mutables, snap->c.mutables, newState, MUTABLES1_MASK_STATE, MUTABLES1_OFFSET_STATE);\r\n return newState;\r\n }\r\n }\r\n \r\n MUTABLES1_WRITE_BIT(ptr->c.mutables, snap->c.mutables, MUTABLES1_MASK_ALLFROZEN);\r\n SOFTWARE_BARRIER;\r\n for (int i=1; ic.numberOfNodesToFreeze; ++i) {\r\n if (snap->c.nodes[i]->isLeaf()) continue; // do not mark leaves\r\n snap->c.nodes[i]->marked = true; // finalize all but first node\r\n }\r\n\r\n // CAS in the new sub-tree (update CAS)\r\n rqProvider->linearize_update_at_cas(tid, snap->c.field, snap->c.nodes[1], snap->c.newNode, snap->c.insertedNodes, snap->c.deletedNodes);\r\n// __sync_bool_compare_and_swap(snap->c.field, snap->c.nodes[1], snap->c.newNode);\r\n TRACE COUTATOMICTID((helpingOther?\" \":\"\")<<\"help CAS'ed to newNode@0x\"<<((uintptr_t)snap->c.newNode)<c.mutables, snap->c.mutables, SCXRecord::STATE_COMMITTED, MUTABLES1_MASK_STATE, MUTABLES1_OFFSET_STATE);\r\n \r\n TRACE COUTATOMICTID((helpingOther?\" \":\"\")<<\"help return COMMITTED after performing update cas\"<::STATE_COMMITTED; // success\r\n}\r\n\r\n#endif\t/* ABTREE_IMPL_H */\r\n" }, { "path": "datastructures/trevor_brown_abtree/minimal_example.cpp", "content": "/**\r\n * Author: Trevor Brown (me [at] tbrown [dot] pro).\r\n * Copyright 2018.\r\n * \r\n * This program is free software; you can redistribute it and/or\r\n * modify it under the terms of the GNU General Public License\r\n * as published by the Free Software Foundation, version 2\r\n * of the License.\r\n *\r\n * This program is distributed in the hope that it will be useful,\r\n * but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n * GNU General Public License for more details.\r\n */\r\n\r\n#include \r\n#include \r\n#include \r\n\r\n#include \"brown_ext_abtree_lf_adapter.h\"\r\n\r\nint main(int argc, char** argv) {\r\n\r\n const int NODE_DEGREE = 16;\r\n const int ANY_KEY = 0;\r\n const int NUM_THREADS = 1;\r\n \r\n auto tree = new ds_adapter(NUM_THREADS, ANY_KEY);\r\n \r\n const int threadID = 0;\r\n\r\n tree->initThread(threadID);\r\n \r\n void * oldVal = tree->insertIfAbsent(threadID, 7, (void *) 1020);\r\n assert(oldVal == tree->getNoValue());\r\n \r\n bool result = tree->contains(threadID, 7);\r\n assert(result);\r\n \r\n result = tree->contains(threadID, 8);\r\n assert(!result);\r\n \r\n void * val = tree->find(threadID, 7);\r\n assert(val == (void *) 1020);\r\n \r\n val = tree->erase(threadID, 7);\r\n assert(val == (void *) 1020);\r\n \r\n result = tree->contains(threadID, 7);\r\n assert(!result);\r\n \r\n tree->deinitThread(threadID);\r\n \r\n delete tree;\r\n \r\n std::cout<<\"Passed quick tests.\"<\r\n#include \r\n#include \r\n#include \r\n#include \"common/ThreadRegistry.hpp\"\r\n#include \"ds/natarajan_ext_bst_lf/natarajan_ext_bst_lf_adapter.h\"\r\n\r\n/*\r\n * This is a wrapper to Trevor Brown's implementation of Naratajan's lock=free Tree so we can use it in our benchmarks\r\n */\r\n\r\ntemplate\r\nclass TrevorBrownNatarajanTree {\r\n const int NUM_THREADS = 128;\r\n //ds_adapter>* tree;\r\n ds_adapter* tree;\r\n\r\npublic:\r\n TrevorBrownNatarajanTree(int numThreads) {\r\n const int minValue = 0;\r\n const int maxValue = std::numeric_limits::max();\r\n const int noValue = -1;\r\n //tree = new ds_adapter>(minValue, maxValue, noValue, NUM_THREADS);\r\n tree = new ds_adapter(minValue, maxValue, noValue, NUM_THREADS);\r\n }\r\n\r\n ~TrevorBrownNatarajanTree() {\r\n // TODO: deinit threads?\r\n delete tree;\r\n }\r\n\r\n // Inserts a key only if it's not already present\r\n bool add(K key, const int tid=0) {\r\n int threadID = tl_tcico.tid;\r\n if (threadID == ThreadCheckInCheckOut::NOT_ASSIGNED) {\r\n threadID = ThreadRegistry::getTID();\r\n tree->initThread(threadID);\r\n }\r\n return tree->insertIfAbsent(threadID, key, 1) != tree->getNoValue();\r\n }\r\n\r\n // Returns true only if the key was present\r\n bool remove(K key, const int tid=0) {\r\n int threadID = tl_tcico.tid;\r\n if (threadID == ThreadCheckInCheckOut::NOT_ASSIGNED) {\r\n threadID = ThreadRegistry::getTID();\r\n tree->initThread(threadID);\r\n }\r\n return tree->erase(threadID, key) != tree->getNoValue();\r\n }\r\n\r\n bool contains(K key, const int tid=0) {\r\n int threadID = tl_tcico.tid;\r\n if (threadID == ThreadCheckInCheckOut::NOT_ASSIGNED) {\r\n threadID = ThreadRegistry::getTID();\r\n tree->initThread(threadID);\r\n }\r\n return tree->contains(threadID, key);\r\n }\r\n\r\n // This is not fully transactionally but it's ok because we use it only on initialization.\r\n // We could make it fully transactionally, but we would have to increase the size of allocation/store logs.\r\n void addAll(K** keys, int size, const int tid=0) {\r\n for (int i = 0; i < size; i++) add(*keys[i], tid);\r\n }\r\n\r\n static std::string className() { return \"TrevorBrown-Natarajan-Tree\"; }\r\n\r\n};\r\n\r\n#endif // _TREVOR_BROWN_NATARAJAN_TREE_HP_H_\r\n" }, { "path": "datastructures/trevor_brown_natarajan/ds/natarajan_ext_bst_lf/natarajan_ext_bst_lf_adapter.h", "content": "/* \r\n * Implementation of the lock-free tree of Natarajan and Mittal.\r\n * \r\n * Heavily edited by Trevor Brown (me [at] tbrown [dot] pro).\r\n * (Late 2017, early 2018.)\r\n * \r\n * Notable changes:\r\n * - Converted original implementation to a class.\r\n * - Fixed a bug: atomic_ops types don't contain \"volatile,\" so the original\r\n * implementation behaved erroneously under high contention.\r\n * - Fixed the original implementation's erroneous memory reclamation,\r\n * which would leak many nodes.\r\n * \r\n * This program is free software; you can redistribute it and/or\r\n * modify it under the terms of the GNU General Public License\r\n * as published by the Free Software Foundation, version 2\r\n * of the License.\r\n *\r\n * This program is distributed in the hope that it will be useful,\r\n * but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n * GNU General Public License for more details.\r\n *\r\n * Created on August 31, 2017, 6:22 PM\r\n */\r\n\r\n#ifndef NATARAJAN_EXT_BST_LF_ADAPTER_H\r\n#define NATARAJAN_EXT_BST_LF_ADAPTER_H\r\n\r\n#include \r\n#include \"errors.h\"\r\n#include \"natarajan_ext_bst_lf_stage2_impl.h\"\r\n\r\n#define RECORD_MANAGER_T record_manager>\r\n#define DATA_STRUCTURE_T natarajan_ext_bst_lf\r\n\r\ntemplate , class Alloc = allocator_new, class Pool = pool_none>\r\nclass ds_adapter {\r\nprivate:\r\n const V NO_VALUE;\r\n DATA_STRUCTURE_T * const tree;\r\n\r\npublic:\r\n ds_adapter(const K& MIN_KEY, const K& MAX_KEY, const V& _NO_VALUE, const int numThreads)\r\n : NO_VALUE(_NO_VALUE)\r\n , tree(new DATA_STRUCTURE_T(MAX_KEY, NO_VALUE, numThreads))\r\n {}\r\n ~ds_adapter() {\r\n delete tree;\r\n }\r\n \r\n V getNoValue() {\r\n return NO_VALUE;\r\n }\r\n \r\n void initThread(const int tid) {\r\n tree->initThread(tid);\r\n }\r\n void deinitThread(const int tid) {\r\n tree->deinitThread(tid);\r\n }\r\n\r\n bool contains(const int tid, const K& key) {\r\n return tree->find(tid, key) != getNoValue();\r\n }\r\n V insert(const int tid, const K& key, const V& val) {\r\n error(\"insert-replace not implemented for this data structure\");\r\n }\r\n V insertIfAbsent(const int tid, const K& key, const V& val) {\r\n return tree->insertIfAbsent(tid, key, val);\r\n }\r\n V erase(const int tid, const K& key) {\r\n return tree->erase(tid, key);\r\n }\r\n V find(const int tid, const K& key) {\r\n return tree->find(tid, key);\r\n }\r\n int rangeQuery(const int tid, const K& lo, const K& hi, K * const resultKeys, V * const resultValues) {\r\n error(\"rangeQuery not implemented for this data structure\");\r\n }\r\n /**\r\n * Sequential operation to get the number of keys in the set\r\n */\r\n int getSize() {\r\n return tree->getSize();\r\n }\r\n void printSummary() {\r\n tree->printSummary();\r\n }\r\n long long getKeyChecksum() {\r\n return tree->getKeyChecksum();\r\n }\r\n bool validateStructure() {\r\n return tree->validateStructure();\r\n }\r\n void printObjectSizes() {\r\n std::cout<<\"sizes: node=\"\r\n <<(sizeof(node_t))\r\n <\r\n * Description:\r\n * A Lock Free Binary Search Tree\r\n *\r\n * Copyright (c) 2013-2014.\r\n *\r\n * This program is free software; you can redistribute it and/or\r\n * modify it under the terms of the GNU General Public License\r\n * as published by the Free Software Foundation, version 2\r\n * of the License.\r\n *\r\n * This program is distributed in the hope that it will be useful,\r\n * but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n * GNU General Public License for more details.\r\n\r\nPlease cite our PPoPP 2014 paper - Fast Concurrent Lock-Free Binary Search Trees by Aravind Natarajan and Neeraj Mittal if you use our code in your experiments\t\r\n\r\nFeatures:\r\n1. Insert operations directly install their window without injecting the operation into the tree. They help any conflicting operation at the injection point, \r\nbefore executing their window txn.\r\n2. Delete operations are the same as that of the original algorithm.\r\n \r\n */\r\n\r\n/* \r\n * File: wfrbt.h\r\n * Author: Maya Arbel-Raviv\r\n *\r\n * Created on June 8, 2017, 10:45 AM\r\n */\r\n\r\n/*\r\n * Heavily edited by Trevor Brown (me [at] tbrown [dot] pro).\r\n * (Late 2017, early 2018.)\r\n * \r\n * - Converted to a class and added proper memory reclamation.\r\n * - Fixed a bug: atomic_ops types don't contain \"volatile,\" so the original\r\n * implementation behaved erroneously under high contention.\r\n * - Fixed the original implementation's erroneous memory reclamation,\r\n * which would leak many nodes.\r\n */\r\n\r\n#ifndef NATARAJAN_EXT_BST_LF_H\r\n#define NATARAJAN_EXT_BST_LF_H\r\n\r\n#include \"errors.h\"\r\n#include \"record_manager.h\"\r\n#include \"atomic_ops.h\"\r\n\r\n#if (INDEX_STRUCT == IDX_NATARAJAN_EXT_BST_LF) \r\n#elif (INDEX_STRUCT == IDX_NATARAJAN_EXT_BST_LF_BASELINE)\r\n#error cannot support baseline with int keys and no value. \r\n#else\r\n#error\r\n#endif\r\n\r\n// Most of these macros are not used in this algorithm\r\n\r\n#define MARK_BIT 1\r\n#define FLAG_BIT 0\r\n\r\n#define atomic_cas_full(addr, old_val, new_val) __sync_bool_compare_and_swap(addr, old_val, new_val);\r\n#define create_child_word(addr, mark, flag) (((uintptr_t) addr << 2) + (mark << 1) + (flag))\r\n#define is_marked(x) ( ((x >> 1) & 1) == 1 ? true:false)\r\n#define is_flagged(x) ( (x & 1 ) == 1 ? true:false)\r\n#define get_addr(x) (x >> 2)\r\n#define add_mark_bit(x) (x + 4UL)\r\n#define is_free(x) (((x) & 3) == 0? true:false)\r\n\r\nenum {\r\n INSERT, DELETE\r\n};\r\n\r\nenum {\r\n UNMARK, MARK\r\n};\r\n\r\nenum {\r\n UNFLAG, FLAG\r\n};\r\n\r\ntypedef uintptr_t Word;\r\n\r\ntemplate \r\nstruct node_t {\r\n union {\r\n struct {\r\n skey_t key;\r\n sval_t value;\r\n volatile AO_double_t child;\r\n };\r\n#ifdef MIN_NODE_SIZE\r\n char bytes[MIN_NODE_SIZE];\r\n#endif\r\n };\r\n};\r\n\r\n\r\ntemplate \r\nstruct seekRecord_t {\r\n skey_t leafKey;\r\n sval_t leafValue;\r\n struct node_t* leaf;\r\n struct node_t* parent;\r\n AO_t pL;\r\n bool isLeftL; // is L the left child of P?\r\n struct node_t* lum;\r\n AO_t lumC;\r\n bool isLeftUM; // is last unmarked node's child on access path the left child of the last unmarked node?\r\n};\r\n\r\ntemplate \r\nstruct thread_data_t {\r\n int id;\r\n struct node_t* rootOfTree;\r\n seekRecord_t* sr; // seek record\r\n seekRecord_t * ssr; // secondary seek record\r\n};\r\n\r\n//static __thread thread_data_t * data = NULL;\r\n\r\ntemplate >\r\nclass natarajan_ext_bst_lf {\r\nprivate:\r\n RecMgr * const recmgr;\r\n Compare cmp;\r\n node_t * root;\r\n\r\n seekRecord_t* insseek(thread_data_t* data, skey_t key, int op);\r\n seekRecord_t* delseek(thread_data_t* data, skey_t key, int op);\r\n seekRecord_t* secondary_seek(thread_data_t* data, skey_t key, seekRecord_t* sr);\r\n sval_t delete_node(thread_data_t* data, skey_t key);\r\n sval_t insertIfAbsent(thread_data_t* data, skey_t key, sval_t value);\r\n sval_t search(thread_data_t* data, skey_t key);\r\n int help_conflicting_operation (thread_data_t* data, seekRecord_t* R);\r\n int inject(thread_data_t* data, seekRecord_t* R, int op);\r\n int perform_one_delete_window_operation(thread_data_t* data, seekRecord_t* R, skey_t key);\r\n int perform_one_insert_window_operation(thread_data_t* data, seekRecord_t* R, skey_t newKey, sval_t value);\r\n\r\n void retireDeletedNodes(thread_data_t* data, node_t * node, node_t * targetNode, bool pointerFlagged = false);\r\n \r\n int init[MAX_TID_POW2] = {0,};\r\npublic:\r\n const skey_t MAX_KEY;\r\n const sval_t NO_VALUE;\r\n const int NUM_PROCESSES;\r\n\r\n natarajan_ext_bst_lf(const skey_t& _MAX_KEY, const sval_t& _NO_VALUE, const int numProcesses)\r\n : MAX_KEY(_MAX_KEY)\r\n , NO_VALUE(_NO_VALUE)\r\n , NUM_PROCESSES(numProcesses)\r\n , recmgr(new RecMgr(numProcesses, SIGQUIT)) {\r\n const int tid = 0;\r\n initThread(tid);\r\n \r\n cmp = Compare();\r\n\r\n recmgr->enterQuiescentState(tid); // block crash recovery signal for this thread, and enter an initial quiescent state.\r\n\r\n root = recmgr->template allocate>(tid);\r\n node_t * newLC = recmgr->template allocate>(tid);\r\n node_t * newRC = recmgr->template allocate>(tid);\r\n\r\n memset(newLC, 0, sizeof (struct node_t));\r\n memset(newRC, 0, sizeof (struct node_t));\r\n\r\n root->key = _MAX_KEY;\r\n newLC->key = _MAX_KEY - 1;\r\n newRC->key = _MAX_KEY;\r\n\r\n root->value = NO_VALUE;\r\n newLC->value = NO_VALUE;\r\n newRC->value = NO_VALUE;\r\n\r\n root->child.AO_val1 = create_child_word(newLC, UNMARK, UNFLAG);\r\n root->child.AO_val2 = create_child_word(newRC, UNMARK, UNFLAG);\r\n }\r\n\r\n void freeSubtree(node_t * curr) {\r\n const int tid = 0;\r\n if (curr == NULL) return;\r\n node_t * left = get_left(curr);\r\n node_t * right = get_right(curr);\r\n recmgr->deallocate(tid, curr);\r\n freeSubtree(left);\r\n freeSubtree(right);\r\n }\r\n \r\n ~natarajan_ext_bst_lf() {\r\n freeSubtree(root);\r\n delete recmgr;\r\n }\r\n\r\n void initThread(const int tid) {\r\n if (init[tid]) return; else init[tid] = !init[tid];\r\n\r\n recmgr->initThread(tid);\r\n }\r\n\r\n void deinitThread(const int tid) {\r\n if (!init[tid]) return; else init[tid] = !init[tid];\r\n\r\n recmgr->deinitThread(tid);\r\n }\r\n\r\n sval_t insertIfAbsent(const int tid, skey_t key, sval_t item) { \r\n assert(cmp(key, MAX_KEY-1));\r\n thread_data_t data; \r\n seekRecord_t sr; \r\n seekRecord_t ssr; \r\n data.id = tid;\r\n data.sr = &sr;\r\n data.ssr = &ssr;\r\n data.rootOfTree = root; \r\n return insertIfAbsent(&data,key,item);\r\n }\r\n\r\n sval_t erase(const int tid, skey_t key) { \r\n assert(cmp(key, MAX_KEY-1));\r\n thread_data_t data; \r\n seekRecord_t sr; \r\n seekRecord_t ssr; \r\n data.id = tid;\r\n data.sr = &sr;\r\n data.ssr = &ssr;\r\n data.rootOfTree = root;\r\n return delete_node(&data,key);\r\n }\r\n\r\n sval_t find(const int tid, skey_t key) {\r\n thread_data_t data; \r\n seekRecord_t sr; \r\n seekRecord_t ssr; \r\n data.id = tid;\r\n data.sr = &sr;\r\n data.ssr = &ssr;\r\n data.rootOfTree = root;\r\n return search(&data,key);\r\n }\r\n\r\n node_t * get_root() {\r\n return root;\r\n }\r\n\r\n node_t * get_left(node_t * curr) {\r\n return (node_t *)get_addr(curr->child.AO_val1); \r\n }\r\n\r\n node_t * get_right(node_t * curr) {\r\n return (node_t *)get_addr(curr->child.AO_val2); \r\n }\r\n \r\n long long getKeyChecksum(node_t * curr) {\r\n if (curr == NULL) return 0;\r\n node_t * left = get_left(curr);\r\n node_t * right = get_right(curr);\r\n if (!left && !right) return (long long) curr->key; // leaf\r\n return getKeyChecksum(left) + getKeyChecksum(right);\r\n }\r\n \r\n long long getKeyChecksum() {\r\n return getKeyChecksum(get_left(get_left(root)));\r\n }\r\n \r\n long long getSize(node_t * curr) {\r\n if (curr == NULL) return 0;\r\n node_t * left = get_left(curr);\r\n node_t * right = get_right(curr);\r\n if (!left && !right) return 1; // leaf\r\n return getSize(left) + getSize(right);\r\n }\r\n \r\n bool validateStructure() {\r\n return true;\r\n }\r\n \r\n long long getSize() {\r\n return getSize(get_left(get_left(root)));\r\n }\r\n \r\n long long getSizeInNodes(node_t * const curr) {\r\n if (curr == NULL) return 0;\r\n return 1 + getSizeInNodes(get_left(curr))\r\n + getSizeInNodes(get_right(curr));\r\n }\r\n \r\n long long getSizeInNodes() {\r\n return getSizeInNodes(root);\r\n } \r\n\r\n void printSummary() {\r\n stringstream ss;\r\n ss<printStatus();\r\n }\r\n};\r\n#endif /* NATARAJAN_EXT_BST_LF_H */\r\n" }, { "path": "datastructures/trevor_brown_natarajan/ds/natarajan_ext_bst_lf/natarajan_ext_bst_lf_stage2_impl.h", "content": "/*A Lock Free Binary Search Tree\r\n \r\n * File:\r\n * wfrbt.cpp\r\n * Author(s):\r\n * Aravind Natarajan \r\n * Description:\r\n * A Lock Free Binary Search Tree\r\n *\r\n * Copyright (c) 2013-2014.\r\n *\r\n * This program is free software; you can redistribute it and/or\r\n * modify it under the terms of the GNU General Public License\r\n * as published by the Free Software Foundation, version 2\r\n * of the License.\r\n *\r\n * This program is distributed in the hope that it will be useful,\r\n * but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n * GNU General Public License for more details.\r\n\r\nPlease cite our PPoPP 2014 paper - Fast Concurrent Lock-Free Binary Search Trees by Aravind Natarajan and Neeraj Mittal if you use our code in your experiments\t\r\n\r\nFeatures:\r\n1. Insert operations directly install their window without injecting the operation into the tree. They help any conflicting operation at the injection point, \r\nbefore executing their window txn.\r\n2. Delete operations are the same as that of the original algorithm.\r\n \r\n */\r\n\r\n/* \r\n * File: wfrbt_impl.h\r\n * Author: Maya Arbel-Raviv\r\n *\r\n * Created on June 8, 2017, 10:45 AM\r\n */\r\n\r\n/*\r\n * Heavily edited by Trevor Brown (me [at] tbrown [dot] pro).\r\n * (Late 2017, early 2018.)\r\n * \r\n * - Converted to a class and added proper memory reclamation.\r\n * - Fixed a bug: atomic_ops types don't contain \"volatile,\" so the original\r\n * implementation behaved erroneously under high contention.\r\n * - Fixed the original implementation's erroneous memory reclamation,\r\n * which would leak many nodes.\r\n */\r\n\r\n#ifndef NATARAJAN_EXT_BST_LF_IMPL_H\r\n#define NATARAJAN_EXT_BST_LF_IMPL_H\r\n\r\n#include \"natarajan_ext_bst_lf_stage1.h\"\r\n\r\nstatic inline bool SetBit(volatile size_t *array, int bit) {\r\n bool flag;\r\n __asm__ __volatile__(\"lock bts %2,%1; setb %0\" : \"=q\" (flag) : \"m\" (*array), \"r\" (bit));\r\n return flag;\r\n}\r\n\r\nstatic bool mark_Node(volatile AO_t * word) {\r\n return (SetBit(word, MARK_BIT));\r\n}\r\n\r\nstatic volatile AO_t stop = 0;\r\nstatic volatile AO_t stop2 = 0;\r\n\r\n//long total_insert = 0;\r\n\r\n/* STRUCTURES */\r\nenum {\r\n Front, Back\r\n};\r\n\r\n//long blackCount = -1;\r\n//long leafNodes = 0;\r\n\r\ntemplate \r\nseekRecord_t* natarajan_ext_bst_lf::insseek(thread_data_t* data, skey_t key, int op) {\r\n\r\n node_t * gpar = NULL; // last node (ancestor of parent on access path) whose child pointer field is unmarked\r\n node_t * par = data->rootOfTree;\r\n node_t * leaf;\r\n node_t * leafchild;\r\n\r\n\r\n AO_t parentPointerWord = (size_t) NULL; // contents in gpar\r\n AO_t leafPointerWord = par->child.AO_val1; // contents in par. Tree has two imaginary keys \\inf_{1} and \\inf_{2} which are larger than all other keys. \r\n AO_t leafchildPointerWord; // contents in leaf\r\n\r\n bool isparLC = false; // is par the left child of gpar\r\n bool isleafLC = true; // is leaf the left child of par\r\n bool isleafchildLC; // is leafchild the left child of leaf\r\n\r\n\r\n leaf = (node_t *)get_addr(leafPointerWord);\r\n if (cmp(key, leaf->key)) {\r\n leafchildPointerWord = leaf->child.AO_val1;\r\n isleafchildLC = true;\r\n\r\n } else {\r\n leafchildPointerWord = leaf->child.AO_val2;\r\n isleafchildLC = false;\r\n }\r\n\r\n leafchild = (node_t *)get_addr(leafchildPointerWord);\r\n\r\n\r\n\r\n while (leafchild != NULL) {\r\n\r\n\r\n\r\n if (!is_marked(leafPointerWord)) {\r\n gpar = par;\r\n parentPointerWord = leafPointerWord;\r\n isparLC = isleafLC;\r\n }\r\n\r\n par = leaf;\r\n leafPointerWord = leafchildPointerWord;\r\n isleafLC = isleafchildLC;\r\n\r\n leaf = leafchild;\r\n\r\n\r\n if (cmp(key, leaf->key)) {\r\n leafchildPointerWord = leaf->child.AO_val1;\r\n isleafchildLC = true;\r\n } else {\r\n leafchildPointerWord = leaf->child.AO_val2;\r\n isleafchildLC = false;\r\n }\r\n\r\n leafchild = (node_t *)get_addr(leafchildPointerWord);\r\n\r\n }\r\n\r\n// if (key == leaf->key) {\r\n// // key matches that being inserted\t\r\n// return NULL;\r\n// }\r\n\r\n seekRecord_t* R = data->sr;\r\n R->leafKey = leaf->key;\r\n R->leafValue = leaf->value;\r\n R->parent = par;\r\n R->pL = leafPointerWord;\r\n R->isLeftL = isleafLC;\r\n R->lum = gpar;\r\n R->lumC = parentPointerWord;\r\n R->isLeftUM = isparLC;\r\n return R;\r\n}\r\n\r\ntemplate \r\nseekRecord_t* natarajan_ext_bst_lf::delseek(thread_data_t* data, skey_t key, int op) {\r\n node_t * gpar = NULL; // last node (ancestor of parent on access path) whose child pointer field is unmarked\r\n node_t * par = data->rootOfTree;\r\n node_t * leaf;\r\n node_t * leafchild;\r\n\r\n\r\n AO_t parentPointerWord = (AO_t) NULL; // contents in gpar\r\n AO_t leafPointerWord = par->child.AO_val1; // contents in par. Tree has two imaginary keys \\inf_{1} and \\inf_{2} which are larger than all other keys. \r\n AO_t leafchildPointerWord; // contents in leaf\r\n\r\n bool isparLC = false; // is par the left child of gpar\r\n bool isleafLC = true; // is leaf the left child of par\r\n bool isleafchildLC; // is leafchild the left child of leaf\r\n\r\n\r\n leaf = (node_t *)get_addr(leafPointerWord);\r\n if (cmp(key, leaf->key)) {\r\n leafchildPointerWord = leaf->child.AO_val1;\r\n isleafchildLC = true;\r\n\r\n } else {\r\n leafchildPointerWord = leaf->child.AO_val2;\r\n isleafchildLC = false;\r\n }\r\n\r\n leafchild = (node_t *)get_addr(leafchildPointerWord);\r\n\r\n\r\n\r\n while (leafchild != NULL) {\r\n\r\n\r\n\r\n if (!is_marked(leafPointerWord)) {\r\n gpar = par;\r\n parentPointerWord = leafPointerWord;\r\n isparLC = isleafLC;\r\n }\r\n\r\n par = leaf;\r\n leafPointerWord = leafchildPointerWord;\r\n isleafLC = isleafchildLC;\r\n\r\n leaf = leafchild;\r\n\r\n\r\n if (cmp(key, leaf->key)) {\r\n leafchildPointerWord = leaf->child.AO_val1;\r\n isleafchildLC = true;\r\n } else {\r\n leafchildPointerWord = leaf->child.AO_val2;\r\n isleafchildLC = false;\r\n }\r\n\r\n leafchild = (node_t *)get_addr(leafchildPointerWord);\r\n\r\n }\r\n\r\n // op = DELETE\r\n if (key != leaf->key) {\r\n // key is not found in the tree.\r\n return NULL;\r\n }\r\n\r\n seekRecord_t* R = data->sr;\r\n R->leafKey = leaf->key;\r\n R->leafValue = leaf->value;\r\n R->parent = par;\r\n R->leaf = leaf;\r\n R->pL = leafPointerWord;\r\n R->isLeftL = isleafLC;\r\n R->lum = gpar;\r\n R->lumC = parentPointerWord;\r\n R->isLeftUM = isparLC;\r\n\r\n\r\n\r\n return R;\r\n}\r\n\r\ntemplate \r\nseekRecord_t* natarajan_ext_bst_lf::secondary_seek(thread_data_t* data, skey_t key, seekRecord_t* sr) {\r\n\r\n //std::cout << \"sseek\" << std::endl;\r\n node_t * flaggedLeaf = (node_t *)get_addr(sr->pL);\r\n node_t * gpar = NULL; // last node (ancestor of parent on access path) whose child pointer field is unmarked\r\n node_t * par = data->rootOfTree;\r\n node_t * leaf;\r\n node_t * leafchild;\r\n\r\n\r\n AO_t parentPointerWord = (AO_t) NULL; // contents in gpar\r\n AO_t leafPointerWord = par->child.AO_val1; // contents in par. Tree has two imaginary keys \\inf_{1} and \\inf_{2} which are larger than all other keys. \r\n AO_t leafchildPointerWord; // contents in leaf\r\n\r\n bool isparLC = false; // is par the left child of gpar\r\n bool isleafLC = true; // is leaf the left child of par\r\n bool isleafchildLC; // is leafchild the left child of leaf\r\n\r\n\r\n leaf = (node_t *)get_addr(leafPointerWord);\r\n if (cmp(key, leaf->key)) {\r\n leafchildPointerWord = leaf->child.AO_val1;\r\n isleafchildLC = true;\r\n\r\n } else {\r\n leafchildPointerWord = leaf->child.AO_val2;\r\n isleafchildLC = false;\r\n }\r\n\r\n leafchild = (node_t *)get_addr(leafchildPointerWord);\r\n\r\n\r\n\r\n while (leafchild != NULL) {\r\n\r\n\r\n\r\n if (!is_marked(leafPointerWord)) {\r\n gpar = par;\r\n parentPointerWord = leafPointerWord;\r\n isparLC = isleafLC;\r\n }\r\n\r\n par = leaf;\r\n leafPointerWord = leafchildPointerWord;\r\n isleafLC = isleafchildLC;\r\n\r\n leaf = leafchild;\r\n\r\n\r\n if (cmp(key, leaf->key)) {\r\n leafchildPointerWord = leaf->child.AO_val1;\r\n isleafchildLC = true;\r\n } else {\r\n leafchildPointerWord = leaf->child.AO_val2;\r\n isleafchildLC = false;\r\n }\r\n\r\n leafchild = (node_t *)get_addr(leafchildPointerWord);\r\n\r\n }\r\n\r\n\r\n\r\n if (!is_flagged(leafPointerWord) || (leaf != flaggedLeaf)) {\r\n // operation has been completed by another process.\r\n return NULL;\r\n }\r\n\r\n seekRecord_t* R = data->ssr;\r\n\r\n R->leafKey = leaf->key;\r\n R->parent = par;\r\n R->pL = leafPointerWord;\r\n R->isLeftL = isleafLC;\r\n R->lum = gpar;\r\n R->lumC = parentPointerWord;\r\n R->isLeftUM = isparLC;\r\n return R;\r\n}\r\n\r\ntemplate \r\nsval_t natarajan_ext_bst_lf::search(thread_data_t* data, skey_t key) {\r\n recmgr->leaveQuiescentState(data->id);\r\n node_t * cur = (node_t *)get_addr(data->rootOfTree->child.AO_val1);\r\n skey_t lastKey = 0; \r\n node_t * lastNode = NULL;\r\n while (cur != NULL) {\r\n lastKey = cur->key;\r\n lastNode = cur;\r\n cur = (cmp(key, lastKey) ? (node_t *)get_addr(cur->child.AO_val1) : (node_t *)get_addr(cur->child.AO_val2));\r\n }\r\n if (key == lastKey) {\r\n recmgr->enterQuiescentState(data->id);\r\n return lastNode->value;\r\n }\r\n recmgr->enterQuiescentState(data->id);\r\n return NO_VALUE;\r\n}\r\n\r\n//-------------------------------------------------------------------------------------------------------------------------------------------------------\r\n//-------------------------------------------------------------------------------------------------------------------------------------------------------\r\n\r\ntemplate \r\nvoid natarajan_ext_bst_lf::retireDeletedNodes(thread_data_t* data, node_t * node, node_t * targetNode, bool pointerFlagged) {\r\n // traverse from node, retiring everything we deleted\r\n // (that is: every leaf pointed to by a flagged pointer,\r\n // and every internal node with a flagged pointer.)\r\n if (node == NULL) return;\r\n if (node == targetNode) return; // we reached the end of the nodes we deleted\r\n if ((node_t *) node->child.AO_val1 == NULL) {\r\n // node is a leaf\r\n if (pointerFlagged) {\r\n recmgr->retire(data->id, node);\r\n }\r\n return;\r\n }\r\n // node is internal\r\n if (is_flagged(node->child.AO_val1) || is_flagged(node->child.AO_val2)) {\r\n recmgr->retire(data->id, node);\r\n if (!is_free(node->child.AO_val1)) retireDeletedNodes(data, (node_t *) get_addr(node->child.AO_val1), targetNode, is_flagged(node->child.AO_val1));\r\n if (!is_free(node->child.AO_val2)) retireDeletedNodes(data, (node_t *) get_addr(node->child.AO_val2), targetNode, is_flagged(node->child.AO_val2));\r\n }\r\n}\r\n\r\ntemplate \r\nint natarajan_ext_bst_lf::help_conflicting_operation(thread_data_t* data, seekRecord_t* R) {\r\n int result;\r\n node_t * target = NULL;\r\n if (is_flagged(R->pL)) {\r\n // leaf node is flagged for deletion by another process.\r\n\r\n //1. mark sibling of leaf node for deletion and then read its contents.\r\n\r\n AO_t pS;\r\n\r\n if (R->isLeftL) {\r\n // L is the left child of P\r\n mark_Node(&R->parent->child.AO_val2);\r\n pS = R->parent->child.AO_val2;\r\n\r\n } else {\r\n mark_Node(&R->parent->child.AO_val1);\r\n pS = R->parent->child.AO_val1;\r\n }\r\n\r\n // 2. Execute cas on the last unmarked node to remove the \r\n // if pS is flagged, propagate it. \r\n AO_t newWord;\r\n\r\n if (is_flagged(pS)) {\r\n newWord = create_child_word((node_t *)get_addr(pS), UNMARK, FLAG);\r\n } else {\r\n newWord = create_child_word((node_t *)get_addr(pS), UNMARK, UNFLAG);\r\n }\r\n target = (node_t *) get_addr(pS);\r\n\r\n if (R->isLeftUM) {\r\n result = atomic_cas_full(&R->lum->child.AO_val1, R->lumC, newWord);\r\n } else {\r\n result = atomic_cas_full(&R->lum->child.AO_val2, R->lumC, newWord);\r\n }\r\n\r\n } else {\r\n // leaf node is marked for deletion by another process.\r\n // Note that leaf is not flagged, as it will be taken care of in the above case.\r\n\r\n AO_t newWord;\r\n\r\n if (is_flagged(R->pL)) {\r\n newWord = create_child_word((node_t *)get_addr(R->pL), UNMARK, FLAG);\r\n } else {\r\n newWord = create_child_word((node_t *)get_addr(R->pL), UNMARK, UNFLAG);\r\n }\r\n\r\n target = (node_t *) get_addr(R->pL);\r\n\r\n if (R->isLeftUM) {\r\n result = atomic_cas_full(&R->lum->child.AO_val1, R->lumC, newWord);\r\n } else {\r\n result = atomic_cas_full(&R->lum->child.AO_val2, R->lumC, newWord);\r\n }\r\n }\r\n\r\n if (result) {\r\n retireDeletedNodes(data, (node_t *) get_addr(R->lumC), target);\r\n }\r\n \r\n return result; \r\n}\r\n\r\n//-------------------------------------------------------------------------------------------------------------------------------------------------------\r\n//-------------------------------------------------------------------------------------------------------------------------------------------------------\r\n\r\ntemplate \r\nint natarajan_ext_bst_lf::inject(thread_data_t* data, seekRecord_t* R, int op) {\r\n\r\n // pL is free\t\t\r\n //1. Flag L\t\t\r\n AO_t newWord = create_child_word((node_t *)get_addr(R->pL), UNMARK, FLAG);\r\n int result;\r\n if (R->isLeftL) {\r\n result = atomic_cas_full(&R->parent->child.AO_val1, R->pL, newWord);\r\n\r\n } else {\r\n result = atomic_cas_full(&R->parent->child.AO_val2, R->pL, newWord);\r\n }\r\n return result;\r\n}\r\n\r\ntemplate \r\nsval_t natarajan_ext_bst_lf::insertIfAbsent(thread_data_t* data, skey_t key, sval_t value) {\r\n int injectResult;\r\n// int fasttry = 0;\r\n while (true) {\r\n recmgr->leaveQuiescentState(data->id);\r\n seekRecord_t* R = insseek(data, key, INSERT);\r\n// fasttry++;\r\n if (R->leafKey == key) {\r\n// if (fasttry == 1) {\r\n return R->leafValue;\r\n// } else {\r\n// return NO_VALUE;\r\n// }\r\n }\r\n if (!is_free(R->pL)) {\r\n help_conflicting_operation(data, R);\r\n\r\n recmgr->enterQuiescentState(data->id);\r\n continue;\r\n }\r\n // key not present in the tree. Insert\t\t\r\n injectResult = perform_one_insert_window_operation(data, R, key, value);\r\n if (injectResult == 1) {\r\n // Operation injected and executed\t\t\t\r\n recmgr->enterQuiescentState(data->id);\r\n return NO_VALUE;\r\n }\r\n recmgr->enterQuiescentState(data->id);\r\n }\r\n // execute insert window operation.\t\t\r\n}\r\n\r\ntemplate \r\nsval_t natarajan_ext_bst_lf::delete_node(thread_data_t* data, skey_t key) {\r\n\r\n int injectResult;\r\n sval_t retval = NO_VALUE;\r\n while (true) {\r\n recmgr->leaveQuiescentState(data->id);\r\n seekRecord_t* R = delseek(data, key, DELETE);\r\n if (R == NULL) {\r\n recmgr->enterQuiescentState(data->id);\r\n return retval;\r\n }\r\n // key is present in the tree. Inject operation into the tree\t\t\r\n if (!is_free(R->pL)) {\r\n\r\n help_conflicting_operation(data, R);\r\n\r\n recmgr->enterQuiescentState(data->id);\r\n continue;\r\n }\r\n injectResult = inject(data, R, DELETE);\r\n if (injectResult == 1) {\r\n retval = R->leafValue;\r\n// recmgr->retire(data->id, R->leaf); // if we won consensus and injected the operation, we retire the replaced leaf. (the replaced parent is retired by the guy who marks the sibling pointer in the parent.)\r\n // Operation injected\r\n //data->numActualDelete++;\r\n int res = perform_one_delete_window_operation(data, R, key);\r\n if (res == 1) {\r\n // operation successfully executed.\r\n recmgr->enterQuiescentState(data->id);\r\n return retval;\r\n } else {\r\n // window transaction could not be executed.\r\n // perform secondary seek.\t\t\t\t\r\n while (true) {\r\n R = secondary_seek(data, key, R);\r\n if (R == NULL) {\r\n // flagged leaf not found. Operation has been executed by some other process.\r\n recmgr->enterQuiescentState(data->id);\r\n return retval;\r\n }\r\n res = perform_one_delete_window_operation(data, R, key);\r\n if (res == 1) {\r\n recmgr->enterQuiescentState(data->id);\r\n return retval;\r\n }\r\n }\r\n }\r\n }\r\n recmgr->enterQuiescentState(data->id);\r\n // otherwise, operation was not injected. Restart.\r\n }\r\n}\r\n\r\ntemplate \r\nint natarajan_ext_bst_lf::perform_one_insert_window_operation(thread_data_t* data, seekRecord_t* R, skey_t newKey, sval_t value) {\r\n node_t * newInt;\r\n node_t * newLeaf;\r\n //\t\tif(data->recycledNodes.empty()){\t\t\r\n// node_t * allocedNodeArr = (node_t *)malloc(2 * sizeof (struct node_t)); // new pointerNode_t[2];\r\n// newInt = &allocedNodeArr[0];\r\n// newLeaf = &allocedNodeArr[1];\r\n newInt = recmgr->template allocate>(data->id);\r\n if (newInt == NULL) {\r\n error(\"out of memory\");\r\n }\r\n#ifdef __HANDLE_STATS\r\n GSTATS_APPEND(data->id, node_allocated_addresses, (long long) newInt);\r\n#endif\r\n newLeaf = recmgr->template allocate>(data->id);\r\n if (newLeaf == NULL) {\r\n error(\"out of memory\");\r\n }\r\n#ifdef __HANDLE_STATS\r\n GSTATS_APPEND(data->id, node_allocated_addresses, (long long) newLeaf);\r\n#endif\r\n\r\n /*\t\t}\t\r\n else{\r\n // reuse memory of previously allocated nodes.\r\n newInt = data->recycledNodes.back();\r\n data->recycledNodes.pop_back();\t\t\t\r\n newLeaf = data->recycledNodes.back();\r\n data->recycledNodes.pop_back();\r\n }\r\n */\r\n newLeaf->child.AO_val1 = (size_t) NULL;\r\n newLeaf->child.AO_val2 = (size_t) NULL;\r\n newLeaf->key = newKey;\r\n newLeaf->value = value;\r\n node_t * existLeaf = (node_t *)get_addr(R->pL);\r\n\r\n skey_t existKey = R->leafKey;\r\n\r\n\r\n if (cmp(newKey, existKey)) {\r\n // key is to be inserted on lchild\r\n newInt->key = existKey;\r\n newInt->child.AO_val1 = create_child_word(newLeaf, 0, 0);\r\n newInt->child.AO_val2 = create_child_word(existLeaf, 0, 0);\r\n\r\n } else {\r\n // key is to be inserted on rchild\r\n newInt->key = newKey;\r\n newInt->child.AO_val2 = create_child_word(newLeaf, 0, 0);\r\n newInt->child.AO_val1 = create_child_word(existLeaf, 0, 0);\r\n\r\n }\r\n\r\n // cas to replace window\t\t\r\n AO_t newCasField;\r\n newCasField = create_child_word(newInt, UNMARK, UNFLAG);\r\n int result;\r\n\r\n if (R->isLeftL) {\r\n result = atomic_cas_full(&R->parent->child.AO_val1, R->pL, newCasField);\r\n } else {\r\n result = atomic_cas_full(&R->parent->child.AO_val2, R->pL, newCasField);\r\n }\r\n if (result == 1) {\r\n // successfully inserted.\t\t\t\r\n //data->numInsert++;\r\n return 1;\r\n } else {\r\n // reuse data and pointer nodes\t\t\t\t\r\n recmgr->deallocate(data->id, newInt);\r\n recmgr->deallocate(data->id, newLeaf);\r\n //data->recycledNodes.push_back(newInt);\r\n //data->recycledNodes.push_back(newLeaf);\r\n return 0;\r\n\r\n }\r\n\r\n}\r\n\r\n/*************************************************************************************************/\r\n\r\ntemplate \r\nint natarajan_ext_bst_lf::perform_one_delete_window_operation(thread_data_t* data, seekRecord_t* R, skey_t key) {\r\n // mark sibling.\r\n AO_t pS;\r\n bool markResult = 0;\r\n if (R->isLeftL) {\r\n // L is the left child of P\r\n markResult = mark_Node(&R->parent->child.AO_val2);\r\n pS = R->parent->child.AO_val2;\r\n\r\n } else {\r\n markResult = mark_Node(&R->parent->child.AO_val1);\r\n pS = R->parent->child.AO_val1;\r\n }\r\n //cout<<\"key=\"<leafKey<<\" markResult=\"<parent\r\n// recmgr->retire(data->id, R->parent);\r\n// }\r\n\r\n AO_t newWord;\r\n\r\n if (is_flagged(pS)) {\r\n newWord = create_child_word((node_t *)get_addr(pS), UNMARK, FLAG);\r\n } else {\r\n newWord = create_child_word((node_t *)get_addr(pS), UNMARK, UNFLAG);\r\n }\r\n\r\n int result;\r\n\r\n if (R->isLeftUM) {\r\n result = atomic_cas_full(&R->lum->child.AO_val1, R->lumC, newWord);\r\n } else {\r\n result = atomic_cas_full(&R->lum->child.AO_val2, R->lumC, newWord);\r\n }\r\n \r\n if (result) {\r\n retireDeletedNodes(data, (node_t *) get_addr(R->lumC), (node_t *) get_addr(pS));\r\n }\r\n \r\n return result;\r\n}\r\n\r\n#endif /* NATARAJAN_EXT_BST_LF_IMPL_H */\r\n\r\n" }, { "path": "graphs/BenchmarkLatencyCounter.hpp", "content": "/*\r\n * Copyright 2017-2018\r\n * Andreia Correia \r\n * Pedro Ramalhete \r\n * Pascal Felber \r\n * Nachshon Cohen \r\n *\r\n * This work is published under the MIT license. See LICENSE.txt\r\n */\r\n#ifndef _BENCHMARK_LATENCY_COUNTER_H_\r\n#define _BENCHMARK_LATENCY_COUNTER_H_\r\n\r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \r\n\r\n\r\nusing namespace std;\r\nusing namespace chrono;\r\n\r\n\r\n/**\r\n * This is a micro-benchmark for measuring on an array of counters\r\n */\r\nclass BenchmarkLatencyCounter {\r\n\r\nprivate:\r\n // Latency constants\r\n static const long long kLatencyMeasures = 1000000LL; // We measure 100M iterations\r\n static const long long kLatencyWarmups = 100000LL; // Plus these many warmup\r\n\r\n static const long long NSEC_IN_SEC = 1000000000LL;\r\n static const uint64_t NUM_COUNTERS = 64;\r\n\r\n int numThreads;\r\n\r\npublic:\r\n struct Result {\r\n uint64_t delay50000;\r\n uint64_t delay90000;\r\n uint64_t delay99000;\r\n uint64_t delay99900;\r\n uint64_t delay99990;\r\n uint64_t delay99999;\r\n };\r\n\r\n BenchmarkLatencyCounter(int numThreads) {\r\n this->numThreads = numThreads;\r\n }\r\n\r\n /*\r\n * Execute latency benchmarks\r\n * We only do one run for this benchmark\r\n */\r\n template class TMTYPE>\r\n Result latencyBenchmark(std::string& className) {\r\n atomic start = { false };\r\n TMTYPE *counters;\r\n TM::template updateTx([&] () { // It's ok to pass by reference because we're single-threaded\r\n counters = (TMTYPE*)TM::tmMalloc(sizeof(TMTYPE)*NUM_COUNTERS);\r\n for (int i = 0; i < NUM_COUNTERS; i++) counters[i] = 0;\r\n });\r\n\r\n auto latency_lambda = [this,&start,&counters](nanoseconds* delays, const int tid) {\r\n long long delayIndex = 0;\r\n while (!start.load()) this_thread::yield();\r\n // Warmup + Measurements\r\n for (int iter=0; iter < (kLatencyWarmups+kLatencyMeasures)/numThreads; iter++) {\r\n // Alternate transactions between left-right and right-left\r\n auto startBeats = steady_clock::now();\r\n TM::updateTx([=] () {\r\n for (int i = 0; i < NUM_COUNTERS; i++) counters[i] = counters[i]+1;\r\n });\r\n auto stopBeats = steady_clock::now();\r\n if (iter >= kLatencyWarmups/numThreads) delays[delayIndex++] = (stopBeats-startBeats);\r\n TM::updateTx([=] () {\r\n for (int i = NUM_COUNTERS-1; i > 0; i--) counters[i] = counters[i]+1;\r\n });\r\n }\r\n };\r\n\r\n nanoseconds* delays[numThreads];\r\n for (int it = 0; it < numThreads; it++) {\r\n delays[it] = new nanoseconds[kLatencyMeasures/numThreads];\r\n for (int imeas=0; imeas < kLatencyMeasures/numThreads; imeas++) delays[it][imeas] = 0ns;\r\n }\r\n\r\n cout << \"##### \" << TM::className() << \" ##### \\n\";\r\n className = TM::className();\r\n thread latencyThreads[numThreads];\r\n for (int tid = 0; tid < numThreads; tid++) latencyThreads[tid] = thread(latency_lambda, delays[tid], tid);\r\n start.store(true);\r\n this_thread::sleep_for(50ms);\r\n for (int tid = 0; tid < numThreads; tid++) latencyThreads[tid].join();\r\n\r\n // Aggregate all the delays for enqueues and dequeues and compute the maxs\r\n cout << \"Aggregating delays for \" << kLatencyMeasures/1000000 << \" million measurements...\\n\";\r\n vector aggDelay(kLatencyMeasures);\r\n long long idx = 0;\r\n for (int it = 0; it < numThreads; it++) {\r\n for (int i = 0; i < kLatencyMeasures/numThreads; i++) {\r\n aggDelay[idx] = delays[it][i];\r\n idx++;\r\n }\r\n }\r\n\r\n // Sort the aggregated delays\r\n cout << \"Sorting delays...\\n\";\r\n sort(aggDelay.begin(), aggDelay.end());\r\n\r\n // Show the 50% (median), 90%, 99%, 99.9%, 99.99%, 99.999% and maximum in microsecond/nanoseconds units\r\n long per50000 = (long)(kLatencyMeasures*50000LL/100000LL);\r\n long per70000 = (long)(kLatencyMeasures*70000LL/100000LL);\r\n long per80000 = (long)(kLatencyMeasures*80000LL/100000LL);\r\n long per90000 = (long)(kLatencyMeasures*90000LL/100000LL);\r\n long per99000 = (long)(kLatencyMeasures*99000LL/100000LL);\r\n long per99900 = (long)(kLatencyMeasures*99900LL/100000LL);\r\n long per99990 = (long)(kLatencyMeasures*99990LL/100000LL);\r\n long per99999 = (long)(kLatencyMeasures*99999LL/100000LL);\r\n long imax = kLatencyMeasures-1;\r\n\r\n cout << \"Enqueue delay (us): 50%=\" << aggDelay[per50000].count()/1000 << \" 70%=\" << aggDelay[per70000].count()/1000 << \" 80%=\" << aggDelay[per80000].count()/1000\r\n << \" 90%=\" << aggDelay[per90000].count()/1000 << \" 99%=\" << aggDelay[per99000].count()/1000\r\n << \" 99.9%=\" << aggDelay[per99900].count()/1000 << \" 99.99%=\" << aggDelay[per99990].count()/1000\r\n << \" 99.999%=\" << aggDelay[per99999].count()/1000 << \" max=\" << aggDelay[imax].count()/1000 << \"\\n\";\r\n\r\n Result res = {\r\n (uint64_t)aggDelay[per50000].count()/1000, (uint64_t)aggDelay[per90000].count()/1000,\r\n (uint64_t)aggDelay[per99000].count()/1000, (uint64_t)aggDelay[per99900].count()/1000,\r\n (uint64_t)aggDelay[per99990].count()/1000, (uint64_t)aggDelay[per99999].count()/1000\r\n };\r\n/*\r\n // Show in csv format\r\n cout << \"delay (us):\\n\";\r\n cout << \"50, \" << aggDelay[per50000].count()/1000 << \"\\n\";\r\n cout << \"90, \" << aggDelay[per90000].count()/1000 << \"\\n\";\r\n cout << \"99, \" << aggDelay[per99000].count()/1000 << \"\\n\";\r\n cout << \"99.9, \" << aggDelay[per99900].count()/1000 << \"\\n\";\r\n cout << \"99.99, \" << aggDelay[per99990].count()/1000 << \"\\n\";\r\n cout << \"99.999, \" << aggDelay[per99999].count()/1000 << \"\\n\";\r\n*/\r\n TM::template updateTx([&] () { // It's ok to pass by reference because we're single-threaded\r\n TM::tmFree(counters);\r\n });\r\n\r\n // Cleanup\r\n for (int it = 0; it < numThreads; it++) delete[] delays[it];\r\n return res;\r\n }\r\n\r\n\r\n#ifdef NEVER\r\npublic:\r\n\r\n static void allLatencyTests() {\r\n // Burst Latency benchmarks\r\n //vector threadList = { 30, 30, 30, 30, 30, 30, 30 }; // For the latency table in the paper\r\n //vector threadList = { 4 };\r\n vector threadList = { 1, 2, 4, 8, 12, 16, 20, 24, 28, 30, 32 };\r\n\r\n for (int nThreads : threadList) {\r\n BenchmarkLatencyQ bench(nThreads, 0, 0s); // Only the numThreads is used in this test\r\n std::cout << \"\\n----- Burst Latency numThreads=\" << bench.numThreads << \" kLatencyMeasures=\" << kLatencyMeasures/1000000LL << \"M -----\\n\";\r\n bench.latencyBurstBenchmark>();\r\n }\r\n for (int nThreads : threadList) {\r\n BenchmarkLatencyQ bench(nThreads, 0, 0s); // Only the numThreads is used in this test\r\n std::cout << \"\\n----- Burst Latency numThreads=\" << bench.numThreads << \" kLatencyMeasures=\" << kLatencyMeasures/1000000LL << \"M -----\\n\";\r\n bench.latencyBurstBenchmark>();\r\n }\r\n for (int nThreads : threadList) {\r\n BenchmarkLatencyQ bench(nThreads, 0, 0s); // Only the numThreads is used in this test\r\n std::cout << \"\\n----- Burst Latency numThreads=\" << bench.numThreads << \" kLatencyMeasures=\" << kLatencyMeasures/1000000LL << \"M -----\\n\";\r\n bench.latencyBurstBenchmark>();\r\n }\r\n /*\r\n for (int nThreads : threadList) {\r\n BenchmarkLatencyQ bench(nThreads, 0, 0s); // Only the numThreads is used in this test\r\n std::cout << \"\\n----- Burst Latency numThreads=\" << bench.numThreads << \" kLatencyMeasures=\" << kLatencyMeasures/1000000LL << \"M -----\\n\";\r\n bench.latencyBurstBenchmark>();\r\n }\r\n for (int nThreads : threadList) {\r\n BenchmarkLatencyQ bench(nThreads, 0, 0s); // Only the numThreads is used in this test\r\n std::cout << \"\\n----- Burst Latency numThreads=\" << bench.numThreads << \" kLatencyMeasures=\" << kLatencyMeasures/1000000LL << \"M -----\\n\";\r\n bench.latencyBurstBenchmark>();\r\n }\r\n */\r\n }\r\n#endif\r\n};\r\n\r\n#endif\r\n" }, { "path": "graphs/BenchmarkLatencyQueues.hpp", "content": "/*\r\n * Copyright 2017-2018\r\n * Andreia Correia \r\n * Pedro Ramalhete \r\n * Pascal Felber \r\n * Nachshon Cohen \r\n *\r\n * This work is published under the MIT license. See LICENSE.txt\r\n */\r\n#ifndef _BENCHMARK_LATENCY_Q_H_\r\n#define _BENCHMARK_LATENCY_Q_H_\r\n\r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \r\n\r\n\r\nusing namespace std;\r\nusing namespace chrono;\r\n\r\n\r\n/**\r\n * This is a micro-benchmark for measuring latency\r\n */\r\nclass BenchmarkLatencyQueues {\r\n\r\nprivate:\r\n struct UserData {\r\n long long seq;\r\n int tid;\r\n UserData(long long lseq, int ltid) {\r\n this->seq = lseq;\r\n this->tid = ltid;\r\n }\r\n UserData() {\r\n this->seq = -2;\r\n this->tid = -2;\r\n }\r\n UserData(const UserData &other) : seq(other.seq), tid(other.tid) { }\r\n\r\n bool operator < (const UserData& other) const {\r\n return seq < other.seq;\r\n }\r\n };\r\n\r\n struct Result {\r\n nanoseconds nsEnq = 0ns;\r\n nanoseconds nsDeq = 0ns;\r\n long long numEnq = 0;\r\n long long numDeq = 0;\r\n long long totOpsSec = 0;\r\n\r\n Result() { }\r\n\r\n Result(const Result &other) {\r\n nsEnq = other.nsEnq;\r\n nsDeq = other.nsDeq;\r\n numEnq = other.numEnq;\r\n numDeq = other.numDeq;\r\n totOpsSec = other.totOpsSec;\r\n }\r\n\r\n bool operator < (const Result& other) const {\r\n return totOpsSec < other.totOpsSec;\r\n }\r\n };\r\n\r\n // Latency constants\r\n static const long long kLatencyMeasures = 200000000LL; // We measure 100M iterations divided among the different threads\r\n static const long long kLatencyWarmupIterations = 10; // At start of latency tests we do 10M warmup enqueues and dequeues in bursts of 100K\r\n static const long long kLatencyIterations = 200; // We do this many iterations of 100K enqueues and dequeues until we get kLatencyMeasures\r\n static const long long kLatencyBurst = kLatencyMeasures/kLatencyIterations;\r\n\r\n static const long long NSEC_IN_SEC = 1000000000LL;\r\n\r\n int numThreads;\r\n int numRuns;\r\n seconds testLengthSeconds;\r\n\r\npublic:\r\n BenchmarkLatencyQueues(int numThreads, int numRuns, seconds testLength) {\r\n this->numThreads = numThreads;\r\n this->numRuns = numRuns;\r\n this->testLengthSeconds = testLength;\r\n }\r\n\r\n /*\r\n * Execute latency benchmarks\r\n * Make sure to enable high priority for the Windows process\r\n *\r\n * We can use this Mathematica function to compute the Inverse CDF of a Poisson and model the latency at 99.99% for lock-free algorithms:\r\n * https://reference.wolfram.com/language/ref/InverseCDF.html\r\n *\r\n * We only do one run for this benchmark\r\n */\r\n template\r\n void latencyBurstBenchmark() {\r\n atomic startEnq = { false };\r\n atomic startDeq = { false };\r\n atomic barrier = { 0 };\r\n Q* queue = new Q(numThreads);\r\n\r\n auto latency_lambda = [this,&startEnq,&startDeq,&barrier,&queue](nanoseconds* enqDelays, nanoseconds* deqDelays, const int tid) {\r\n UserData ud(0,0);\r\n long long enqDelayIndex = 0;\r\n long long deqDelayIndex = 0;\r\n\r\n // Warmup + Measurements\r\n for (int iter=0; iter < kLatencyIterations+kLatencyWarmupIterations; iter++) {\r\n // Start with enqueues\r\n while (!startEnq.load()) this_thread::yield();\r\n for (long long i = 0; i < kLatencyBurst/numThreads; i++) {\r\n auto startBeats = steady_clock::now();\r\n queue->enqueue(&ud, tid);\r\n auto stopBeats = steady_clock::now();\r\n if (iter >= kLatencyWarmupIterations) enqDelays[enqDelayIndex++] = (stopBeats-startBeats);\r\n }\r\n if (barrier.fetch_add(1) == numThreads) cout << \"ERROR: in barrier\\n\";\r\n // dequeues\r\n while (!startDeq.load()) this_thread::yield();\r\n for (long long i = 0; i < kLatencyBurst/numThreads; i++) {\r\n auto startBeats = steady_clock::now();\r\n if (queue->dequeue(tid) == nullptr) cout << \"ERROR: dequeued nullptr in i=\" << i << \"\\n\";\r\n auto stopBeats = steady_clock::now();\r\n if (iter >= kLatencyWarmupIterations) deqDelays[deqDelayIndex++] = (stopBeats-startBeats);\r\n }\r\n if (barrier.fetch_add(1) == numThreads) cout << \"ERROR: in barrier\\n\";\r\n }\r\n };\r\n\r\n nanoseconds* enqDelays[numThreads]; // Half enqueues and half dequeues\r\n nanoseconds* deqDelays[numThreads];\r\n for (int it = 0; it < numThreads; it++) {\r\n enqDelays[it] = new nanoseconds[kLatencyMeasures/numThreads];\r\n deqDelays[it] = new nanoseconds[kLatencyMeasures/numThreads];\r\n for (int imeas=0; imeas < kLatencyMeasures/numThreads; imeas++) {\r\n enqDelays[it][imeas] = 0ns;\r\n deqDelays[it][imeas] = 0ns;\r\n }\r\n }\r\n\r\n cout << \"##### \" << queue->className() << \" ##### \\n\";\r\n thread latencyThreads[numThreads];\r\n for (int tid = 0; tid < numThreads; tid++) latencyThreads[tid] = thread(latency_lambda, enqDelays[tid], deqDelays[tid], tid);\r\n this_thread::sleep_for(50ms);\r\n for (int iter=0; iter < kLatencyIterations+kLatencyWarmupIterations; iter++) {\r\n // enqueue round\r\n startEnq.store(true);\r\n while (barrier.load() != numThreads) this_thread::yield();\r\n startEnq.store(false);\r\n long tmp = numThreads;\r\n if (!barrier.compare_exchange_strong(tmp, 0)) cout << \"ERROR: CAS\\n\";\r\n // dequeue round\r\n startDeq.store(true);\r\n while (barrier.load() != numThreads) this_thread::yield();\r\n startDeq.store(false);\r\n tmp = numThreads;\r\n if (!barrier.compare_exchange_strong(tmp, 0)) cout << \"ERROR: CAS\\n\";\r\n }\r\n for (int tid = 0; tid < numThreads; tid++) latencyThreads[tid].join();\r\n delete queue;\r\n\r\n // Aggregate all the delays for enqueues and dequeues and compute the maxs\r\n cout << \"Aggregating delays for \" << kLatencyMeasures/1000000 << \" million measurements...\\n\";\r\n vector aggEnqDelay(kLatencyMeasures);\r\n long long idx = 0;\r\n for (int it = 0; it < numThreads; it++) {\r\n for (int i = 0; i < kLatencyMeasures/numThreads; i++) {\r\n aggEnqDelay[idx] = enqDelays[it][i];\r\n idx++;\r\n }\r\n }\r\n vector aggDeqDelay(kLatencyMeasures);\r\n idx = 0;\r\n for (int it = 0; it < numThreads; it++) {\r\n for (int i = 0; i < kLatencyMeasures/numThreads; i++) {\r\n aggDeqDelay[idx] = deqDelays[it][i];\r\n idx++;\r\n }\r\n }\r\n\r\n // Sort the aggregated delays\r\n cout << \"Sorting delays...\\n\";\r\n sort(aggEnqDelay.begin(), aggEnqDelay.end());\r\n sort(aggDeqDelay.begin(), aggDeqDelay.end());\r\n\r\n // Show the 50% (median), 90%, 99%, 99.9%, 99.99%, 99.999% and maximum in microsecond/nanoseconds units\r\n long per50000 = (long)(kLatencyMeasures*50000LL/100000LL);\r\n long per70000 = (long)(kLatencyMeasures*70000LL/100000LL);\r\n long per80000 = (long)(kLatencyMeasures*80000LL/100000LL);\r\n long per90000 = (long)(kLatencyMeasures*90000LL/100000LL);\r\n long per99000 = (long)(kLatencyMeasures*99000LL/100000LL);\r\n long per99900 = (long)(kLatencyMeasures*99900LL/100000LL);\r\n long per99990 = (long)(kLatencyMeasures*99990LL/100000LL);\r\n long per99999 = (long)(kLatencyMeasures*99999LL/100000LL);\r\n long imax = kLatencyMeasures-1;\r\n\r\n cout << \"Enqueue delay (us): 50%=\" << aggEnqDelay[per50000].count()/1000 << \" 70%=\" << aggEnqDelay[per70000].count()/1000 << \" 80%=\" << aggEnqDelay[per80000].count()/1000\r\n << \" 90%=\" << aggEnqDelay[per90000].count()/1000 << \" 99%=\" << aggEnqDelay[per99000].count()/1000\r\n << \" 99.9%=\" << aggEnqDelay[per99900].count()/1000 << \" 99.99%=\" << aggEnqDelay[per99990].count()/1000\r\n << \" 99.999%=\" << aggEnqDelay[per99999].count()/1000 << \" max=\" << aggEnqDelay[imax].count()/1000 << \"\\n\";\r\n cout << \"Dequeue delay (us): 50%=\" << aggDeqDelay[per50000].count()/1000 << \" 70%=\" << aggDeqDelay[per70000].count()/1000 << \" 80%=\" << aggDeqDelay[per80000].count()/1000\r\n << \" 90%=\" << aggDeqDelay[per90000].count()/1000 << \" 99%=\" << aggDeqDelay[per99000].count()/1000\r\n << \" 99.9%=\" << aggDeqDelay[per99900].count()/1000 << \" 99.99%=\" << aggDeqDelay[per99990].count()/1000\r\n << \" 99.999%=\" << aggDeqDelay[per99999].count()/1000 << \" max=\" << aggDeqDelay[imax].count()/1000 << \"\\n\";\r\n\r\n // Show in csv format\r\n cout << \"Enqueue delay (us):\\n\";\r\n cout << \"50, \" << aggEnqDelay[per50000].count()/1000 << \"\\n\";\r\n cout << \"90, \" << aggEnqDelay[per90000].count()/1000 << \"\\n\";\r\n cout << \"99, \" << aggEnqDelay[per99000].count()/1000 << \"\\n\";\r\n cout << \"99.9, \" << aggEnqDelay[per99900].count()/1000 << \"\\n\";\r\n cout << \"99.99, \" << aggEnqDelay[per99990].count()/1000 << \"\\n\";\r\n cout << \"99.999, \" << aggEnqDelay[per99999].count()/1000 << \"\\n\";\r\n cout << \"Dequeue delay (us):\\n\";\r\n cout << \"50, \" << aggDeqDelay[per50000].count()/1000 << \"\\n\";\r\n cout << \"90, \" << aggDeqDelay[per90000].count()/1000 << \"\\n\";\r\n cout << \"99, \" << aggDeqDelay[per99000].count()/1000 << \"\\n\";\r\n cout << \"99.9, \" << aggDeqDelay[per99900].count()/1000 << \"\\n\";\r\n cout << \"99.99, \" << aggDeqDelay[per99990].count()/1000 << \"\\n\";\r\n cout << \"99.999, \" << aggDeqDelay[per99999].count()/1000 << \"\\n\";\r\n\r\n // Cleanup\r\n for (int it = 0; it < numThreads; it++) {\r\n delete[] enqDelays[it];\r\n delete[] deqDelays[it];\r\n }\r\n }\r\n\r\n\r\n#ifdef NEVER\r\npublic:\r\n\r\n static void allLatencyTests() {\r\n // Burst Latency benchmarks\r\n //vector threadList = { 30, 30, 30, 30, 30, 30, 30 }; // For the latency table in the paper\r\n //vector threadList = { 4 };\r\n vector threadList = { 1, 2, 4, 8, 12, 16, 20, 24, 28, 30, 32 };\r\n\r\n for (int nThreads : threadList) {\r\n BenchmarkLatencyQ bench(nThreads, 0, 0s); // Only the numThreads is used in this test\r\n std::cout << \"\\n----- Burst Latency numThreads=\" << bench.numThreads << \" kLatencyMeasures=\" << kLatencyMeasures/1000000LL << \"M -----\\n\";\r\n bench.latencyBurstBenchmark>();\r\n }\r\n for (int nThreads : threadList) {\r\n BenchmarkLatencyQ bench(nThreads, 0, 0s); // Only the numThreads is used in this test\r\n std::cout << \"\\n----- Burst Latency numThreads=\" << bench.numThreads << \" kLatencyMeasures=\" << kLatencyMeasures/1000000LL << \"M -----\\n\";\r\n bench.latencyBurstBenchmark>();\r\n }\r\n for (int nThreads : threadList) {\r\n BenchmarkLatencyQ bench(nThreads, 0, 0s); // Only the numThreads is used in this test\r\n std::cout << \"\\n----- Burst Latency numThreads=\" << bench.numThreads << \" kLatencyMeasures=\" << kLatencyMeasures/1000000LL << \"M -----\\n\";\r\n bench.latencyBurstBenchmark>();\r\n }\r\n /*\r\n for (int nThreads : threadList) {\r\n BenchmarkLatencyQ bench(nThreads, 0, 0s); // Only the numThreads is used in this test\r\n std::cout << \"\\n----- Burst Latency numThreads=\" << bench.numThreads << \" kLatencyMeasures=\" << kLatencyMeasures/1000000LL << \"M -----\\n\";\r\n bench.latencyBurstBenchmark>();\r\n }\r\n for (int nThreads : threadList) {\r\n BenchmarkLatencyQ bench(nThreads, 0, 0s); // Only the numThreads is used in this test\r\n std::cout << \"\\n----- Burst Latency numThreads=\" << bench.numThreads << \" kLatencyMeasures=\" << kLatencyMeasures/1000000LL << \"M -----\\n\";\r\n bench.latencyBurstBenchmark>();\r\n }\r\n */\r\n }\r\n#endif\r\n};\r\n\r\n#endif\r\n" }, { "path": "graphs/BenchmarkMaps.hpp", "content": "/*\r\n * Copyright 2017-2018\r\n * Andreia Correia \r\n * Pedro Ramalhete \r\n * Pascal Felber \r\n * Nachshon Cohen \r\n *\r\n * This work is published under the MIT license. See LICENSE.txt\r\n */\r\n#ifndef _BENCHMARK_MAPS_H_\r\n#define _BENCHMARK_MAPS_H_\r\n\r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \r\n\r\nusing namespace std;\r\nusing namespace chrono;\r\n\r\n// Regular UserData\r\nstruct UserData {\r\n long long seq;\r\n int tid;\r\n UserData(long long lseq, int ltid=0) {\r\n this->seq = lseq;\r\n this->tid = ltid;\r\n }\r\n UserData() {\r\n this->seq = -2;\r\n this->tid = -2;\r\n }\r\n UserData(const UserData &other) : seq(other.seq), tid(other.tid) { }\r\n\r\n bool operator < (const UserData& other) const {\r\n return seq < other.seq;\r\n }\r\n bool operator == (const UserData& other) const {\r\n return seq == other.seq && tid == other.tid;\r\n }\r\n bool operator != (const UserData& other) const {\r\n return seq != other.seq || tid != other.tid;\r\n }\r\n};\r\n\r\n\r\nnamespace std {\r\n template <>\r\n struct hash {\r\n std::size_t operator()(const UserData& k) const {\r\n using std::size_t;\r\n using std::hash;\r\n return (hash()(k.seq)); // This hash has no collisions, which is irealistic\r\n }\r\n };\r\n}\r\n\r\n\r\n/**\r\n * This is a micro-benchmark of sets, used in the CX paper\r\n */\r\nclass BenchmarkMaps {\r\n\r\nprivate:\r\n struct Result {\r\n nanoseconds nsEnq = 0ns;\r\n nanoseconds nsDeq = 0ns;\r\n long long numEnq = 0;\r\n long long numDeq = 0;\r\n long long totOpsSec = 0;\r\n\r\n Result() { }\r\n\r\n Result(const Result &other) {\r\n nsEnq = other.nsEnq;\r\n nsDeq = other.nsDeq;\r\n numEnq = other.numEnq;\r\n numDeq = other.numDeq;\r\n totOpsSec = other.totOpsSec;\r\n }\r\n\r\n bool operator < (const Result& other) const {\r\n return totOpsSec < other.totOpsSec;\r\n }\r\n };\r\n\r\n static const long long NSEC_IN_SEC = 1000000000LL;\r\n\r\n int numThreads;\r\n\r\npublic:\r\n BenchmarkMaps(int numThreads) {\r\n this->numThreads = numThreads;\r\n }\r\n\r\n\r\n /**\r\n * When doing \"updates\" we execute a random removal and if the removal is successful we do an add() of the\r\n * same item immediately after. This keeps the size of the data structure equal to the original size (minus\r\n * MAX_THREADS items at most) which gives more deterministic results.\r\n */\r\n template class S, typename K, typename V>\r\n long long benchmark(const int updateRatio, const seconds testLengthSeconds, const int numRuns, const int numElements, const bool dedicated=false) {\r\n long long ops[numThreads][numRuns];\r\n long long lengthSec[numRuns];\r\n atomic quit = { false };\r\n atomic startFlag = { false };\r\n S* set = nullptr;\r\n#ifdef TINY_STM\r\n stm_init_thread();\r\n //const int tid = 0;\r\n //WRITE_TX_BEGIN\r\n //set = TM_ALLOC();\r\n //WRITE_TX_END\r\n#endif\r\n\r\n // Create all the keys and values in the concurrent set\r\n K** keyarray = new K*[numElements];\r\n for (int i = 0; i < numElements; i++) keyarray[i] = new K(i);\r\n V** valarray = new V*[numElements];\r\n for (int i = 0; i < numElements; i++) valarray[i] = new V(i);\r\n\r\n // Can either be a Reader or a Writer\r\n auto rw_lambda = [&](const int updateRatio, long long *ops, const int tid) {\r\n \tuint64_t accum = 0;\r\n long long numOps = 0;\r\n#ifdef TINY_STM\r\n stm_init_thread();\r\n#endif\r\n while (!startFlag.load()) ; // spin\r\n uint64_t seed = tid+1234567890123456781ULL;\r\n while (!quit.load()) {\r\n seed = randomLong(seed);\r\n int update = seed%1000;\r\n seed = randomLong(seed);\r\n auto ix = (unsigned int)(seed%numElements);\r\n if (update < updateRatio) {\r\n // I'm a Writer\r\n if (set->remove(*keyarray[ix])) {\r\n \tnumOps++;\r\n \tset->put(*keyarray[ix], *valarray[ix]);\r\n }\r\n numOps++;\r\n } else {\r\n \t// I'm a Reader\r\n set->get(*keyarray[ix]);\r\n seed = randomLong(seed);\r\n ix = (unsigned int)(seed%numElements);\r\n set->get(*keyarray[ix]);\r\n numOps+=2;\r\n }\r\n\r\n }\r\n *ops = numOps;\r\n#ifdef TINY_STM\r\n stm_exit_thread();\r\n#endif\r\n };\r\n\r\n for (int irun = 0; irun < numRuns; irun++) {\r\n set = new S();\r\n // Add all the items to the list\r\n set->addAll(keyarray, valarray, numElements);\r\n if (irun == 0) std::cout << \"##### \" << set->className() << \" ##### \\n\";\r\n thread rwThreads[numThreads];\r\n if (dedicated) {\r\n rwThreads[0] = thread(rw_lambda, 1000, &ops[0][irun], 0);\r\n rwThreads[1] = thread(rw_lambda, 1000, &ops[1][irun], 1);\r\n for (int tid = 2; tid < numThreads; tid++) rwThreads[tid] = thread(rw_lambda, updateRatio, &ops[tid][irun], tid);\r\n } else {\r\n for (int tid = 0; tid < numThreads; tid++) rwThreads[tid] = thread(rw_lambda, updateRatio, &ops[tid][irun], tid);\r\n }\r\n this_thread::sleep_for(100ms);\r\n auto startBeats = steady_clock::now();\r\n startFlag.store(true);\r\n // Sleep for testLengthSeconds seconds\r\n this_thread::sleep_for(testLengthSeconds);\r\n quit.store(true);\r\n auto stopBeats = steady_clock::now();\r\n for (int tid = 0; tid < numThreads; tid++) rwThreads[tid].join();\r\n lengthSec[irun] = (stopBeats-startBeats).count();\r\n if (dedicated) {\r\n // We don't account for the write-only operations but we aggregate the values from the two threads and display them\r\n std::cout << \"Mutative transactions per second = \" << (ops[0][irun] + ops[1][irun])*1000000000LL/lengthSec[irun] << \"\\n\";\r\n ops[0][irun] = 0;\r\n ops[1][irun] = 0;\r\n }\r\n quit.store(false);\r\n startFlag.store(false);\r\n // Measure the time the destructor takes to complete and if it's more than 1 second, print it out\r\n auto startDel = steady_clock::now();\r\n#ifdef TINY_STM\r\n WRITE_TX_BEGIN\r\n TM_FREE(set);\r\n WRITE_TX_END\r\n#endif\r\n delete set;\r\n\r\n auto stopDel = steady_clock::now();\r\n if ((startDel-stopDel).count() > NSEC_IN_SEC) {\r\n std::cout << \"Destructor took \" << (startDel-stopDel).count()/NSEC_IN_SEC << \" seconds\\n\";\r\n }\r\n // Compute ops at the end of each run\r\n long long agg = 0;\r\n for (int tid = 0; tid < numThreads; tid++) {\r\n agg += ops[tid][irun]*1000000000LL/lengthSec[irun];\r\n }\r\n }\r\n\r\n for (int i = 0; i < numElements; i++) delete keyarray[i];\r\n delete[] keyarray;\r\n\r\n // Accounting\r\n vector agg(numRuns);\r\n for (int irun = 0; irun < numRuns; irun++) {\r\n for (int tid = 0; tid < numThreads; tid++) {\r\n agg[irun] += ops[tid][irun]*1000000000LL/lengthSec[irun];\r\n }\r\n }\r\n\r\n // Compute the median. numRuns must be an odd number\r\n sort(agg.begin(),agg.end());\r\n auto maxops = agg[numRuns-1];\r\n auto minops = agg[0];\r\n auto medianops = agg[numRuns/2];\r\n auto delta = (long)(100.*(maxops-minops) / ((double)medianops));\r\n // Printed value is the median of the number of ops per second that all threads were able to accomplish (on average)\r\n std::cout << \"Ops/sec = \" << medianops << \" delta = \" << delta << \"% min = \" << minops << \" max = \" << maxops << \"\\n\";\r\n#ifdef TINY_STM\r\n stm_exit_thread();\r\n#endif\r\n return medianops;\r\n }\r\n\r\n\r\n /**\r\n * An imprecise but fast random number generator\r\n */\r\n uint64_t randomLong(uint64_t x) {\r\n x ^= x >> 12; // a\r\n x ^= x << 25; // b\r\n x ^= x >> 27; // c\r\n return x * 2685821657736338717LL;\r\n }\r\n\r\n};\r\n\r\n#endif\r\n" }, { "path": "graphs/BenchmarkQueues.hpp", "content": "/*\r\n * Copyright 2017-2018\r\n * Andreia Correia \r\n * Pedro Ramalhete \r\n * Pascal Felber \r\n * Nachshon Cohen \r\n *\r\n * This work is published under the MIT license. See LICENSE.txt\r\n */\r\n#ifndef _BENCHMARK_Q_H_\r\n#define _BENCHMARK_Q_H_\r\n\r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \r\n\r\n\r\nusing namespace std;\r\nusing namespace chrono;\r\n\r\nstruct UserData {\r\n long long seq;\r\n int tid;\r\n UserData(long long lseq, int ltid) {\r\n this->seq = lseq;\r\n this->tid = ltid;\r\n }\r\n UserData() {\r\n this->seq = -2;\r\n this->tid = -2;\r\n }\r\n UserData(const UserData &other) : seq(other.seq), tid(other.tid) { }\r\n\r\n bool operator < (const UserData& other) const {\r\n return seq < other.seq;\r\n }\r\n};\r\n\r\n\r\n/**\r\n * This is a micro-benchmark to run the tests shown in CRTurnQueue paper\r\n *\r\n *

Performance Benchmarks

\r\n * TODO\r\n *\r\n *\r\n *

Latency Distribution

\r\n *\r\n * - We fire up 28 threads of type LatencyThread;\r\n * - Each thread does alternatively 1000 enqueue() and 1000 dequeue(). All dequeues are non-null;\r\n * - After start, each thread does 1M iterations as warmup.\r\n * - Measurements are done for 4M iterations, that are saved in a local array, 2M enqueue and 2M dequeue;\r\n * -\r\n *\r\n */\r\nclass BenchmarkQueues {\r\n\r\nprivate:\r\n\r\n struct Result {\r\n nanoseconds nsEnq = 0ns;\r\n nanoseconds nsDeq = 0ns;\r\n long long numEnq = 0;\r\n long long numDeq = 0;\r\n long long totOpsSec = 0;\r\n\r\n Result() { }\r\n\r\n Result(const Result &other) {\r\n nsEnq = other.nsEnq;\r\n nsDeq = other.nsDeq;\r\n numEnq = other.numEnq;\r\n numDeq = other.numDeq;\r\n totOpsSec = other.totOpsSec;\r\n }\r\n\r\n bool operator < (const Result& other) const {\r\n return totOpsSec < other.totOpsSec;\r\n }\r\n };\r\n\r\n // Performance benchmark constants\r\n static const long long kNumPairsWarmup = 1000000LL; // Each threads does 1M iterations as warmup\r\n\r\n // Contants for Ping-Pong performance benchmark\r\n static const int kPingPongBatch = 1000; // Each thread starts by injecting 1k items in the queue\r\n\r\n\r\n static const long long NSEC_IN_SEC = 1000000000LL;\r\n\r\n int numThreads;\r\n\r\npublic:\r\n\r\n BenchmarkQueues(int numThreads) {\r\n this->numThreads = numThreads;\r\n }\r\n\r\n\r\n /**\r\n * enqueue-dequeue pairs: in each iteration a thread executes an enqueue followed by a dequeue;\r\n * the benchmark executes 10^8 pairs partitioned evenly among all threads;\r\n */\r\n template\r\n uint64_t enqDeq(std::string& className, const long numPairs, const int numRuns) {\r\n nanoseconds deltas[numThreads][numRuns];\r\n atomic startFlag = { false };\r\n Q* queue = nullptr;\r\n className = Q::className();\r\n cout << \"##### \" << className << \" ##### \\n\";\r\n\r\n auto enqdeq_lambda = [this,&startFlag,&numPairs,&queue](nanoseconds *delta, const int tid) {\r\n UserData ud(0,0);\r\n while (!startFlag.load()) {} // Spin until the startFlag is set\r\n // Warmup phase\r\n for (long long iter = 0; iter < kNumPairsWarmup/numThreads; iter++) {\r\n queue->enqueue(&ud, tid);\r\n if (queue->dequeue(tid) == nullptr) cout << \"Error at warmup dequeueing iter=\" << iter << \"\\n\";\r\n }\r\n // Measurement phase\r\n auto startBeats = steady_clock::now();\r\n for (long long iter = 0; iter < numPairs/numThreads; iter++) {\r\n queue->enqueue(&ud, tid);\r\n if (queue->dequeue(tid) == nullptr) cout << \"Error at measurement dequeueing iter=\" << iter << \"\\n\";\r\n }\r\n auto stopBeats = steady_clock::now();\r\n *delta = stopBeats - startBeats;\r\n };\r\n\r\n for (int irun = 0; irun < numRuns; irun++) {\r\n queue = new Q(numThreads);\r\n thread enqdeqThreads[numThreads];\r\n for (int tid = 0; tid < numThreads; tid++) enqdeqThreads[tid] = thread(enqdeq_lambda, &deltas[tid][irun], tid);\r\n startFlag.store(true);\r\n // Sleep for 2 seconds just to let the threads see the startFlag\r\n this_thread::sleep_for(2s);\r\n for (int tid = 0; tid < numThreads; tid++) enqdeqThreads[tid].join();\r\n startFlag.store(false);\r\n delete (Q*)queue;\r\n }\r\n\r\n // Sum up all the time deltas of all threads so we can find the median run\r\n vector agg(numRuns);\r\n for (int irun = 0; irun < numRuns; irun++) {\r\n agg[irun] = 0ns;\r\n for (int tid = 0; tid < numThreads; tid++) {\r\n agg[irun] += deltas[tid][irun];\r\n }\r\n }\r\n\r\n // Compute the median. numRuns should be an odd number\r\n sort(agg.begin(),agg.end());\r\n auto median = agg[numRuns/2].count()/numThreads; // Normalize back to per-thread time (mean of time for this run)\r\n\r\n cout << \"Total Ops/sec = \" << numPairs*2*NSEC_IN_SEC/median << \"\\n\";\r\n return (numPairs*2*NSEC_IN_SEC/median);\r\n }\r\n\r\n\r\n /**\r\n * Start with only enqueues 100K/numThreads, wait for them to finish, then do only dequeues but only 100K/numThreads\r\n */\r\n template\r\n void burst(std::string& className, uint64_t& resultsEnq, uint64_t& resultsDeq,\r\n const long long burstSize, const int numIters, const int numRuns, const bool isSC=false) {\r\n Result results[numThreads][numRuns];\r\n atomic startEnq = { false };\r\n atomic startDeq = { false };\r\n atomic barrier = { 0 };\r\n Q* queue = nullptr;\r\n\r\n auto burst_lambda = [this,&startEnq,&startDeq,&burstSize,&barrier,&numIters,&isSC,&queue](Result *res, const int tid) {\r\n UserData ud(0,0);\r\n // Warmup only if it is not Single-Consumer\r\n if (!isSC) {\r\n const long long warmupIters = 100000LL; // Do 100K for each thread as a warmup\r\n for (long long iter = 0; iter < warmupIters; iter++) queue->enqueue(&ud, tid);\r\n for (long long iter = 0; iter < warmupIters; iter++) {\r\n if (queue->dequeue(tid) == nullptr) cout << \"ERROR: warmup dequeued nullptr in iter=\" << iter << \"\\n\";\r\n }\r\n }\r\n // Measurements\r\n for (int iter = 0; iter < numIters; iter++) {\r\n // Start with enqueues\r\n while (!startEnq.load()) {} // spin is better than yield here\r\n auto startBeats = steady_clock::now();\r\n for (long long i = 0; i < burstSize/numThreads; i++) {\r\n queue->enqueue(&ud, tid);\r\n }\r\n auto stopBeats = steady_clock::now();\r\n res->nsEnq += (stopBeats-startBeats);\r\n res->numEnq += burstSize/numThreads;\r\n if (barrier.fetch_add(1) == numThreads) cout << \"ERROR: in barrier\\n\";\r\n // dequeues\r\n while (!startDeq.load()) { } // spin is better than yield here\r\n if (isSC) { // Handle the single-consumer case\r\n if (tid == 0) {\r\n startBeats = steady_clock::now();\r\n // We need to deal with rounding errors in the single-consumer case\r\n for (long long i = 0; i < ((long long)(burstSize/numThreads))*numThreads; i++) {\r\n if (queue->dequeue(tid) == nullptr) {\r\n cout << \"ERROR: dequeued nullptr in iter=\" << i << \"\\n\";\r\n assert(false);\r\n }\r\n }\r\n stopBeats = steady_clock::now();\r\n if (queue->dequeue(tid) != nullptr) cout << \"ERROR: dequeued non-null, there must be duplicate items!\\n\";\r\n res->nsDeq += (stopBeats-startBeats);\r\n res->numDeq += burstSize/numThreads;\r\n }\r\n } else {\r\n startBeats = steady_clock::now();\r\n for (long long i = 0; i < burstSize/numThreads; i++) {\r\n if (queue->dequeue(tid) == nullptr) {\r\n cout << \"ERROR: dequeued nullptr in iter=\" << i << \"\\n\";\r\n assert(false);\r\n }\r\n }\r\n stopBeats = steady_clock::now();\r\n res->nsDeq += (stopBeats-startBeats);\r\n res->numDeq += burstSize/numThreads;\r\n }\r\n if (barrier.fetch_add(1) == numThreads) cout << \"ERROR: in barrier\\n\";\r\n }\r\n };\r\n\r\n for (int irun = 0; irun < numRuns; irun++) {\r\n queue = new Q(numThreads);\r\n if (irun == 0) {\r\n className = queue->className();\r\n cout << \"##### \" << queue->className() << \" ##### \\n\";\r\n }\r\n thread burstThreads[numThreads];\r\n for (int tid = 0; tid < numThreads; tid++) burstThreads[tid] = thread(burst_lambda, &results[tid][irun], tid);\r\n this_thread::sleep_for(100ms);\r\n for (int iter=0; iter < numIters; iter++) {\r\n // enqueue round\r\n startEnq.store(true);\r\n while (barrier.load() != numThreads) this_thread::yield();\r\n startEnq.store(false);\r\n long tmp = numThreads;\r\n if (!barrier.compare_exchange_strong(tmp, 0)) cout << \"ERROR: CAS\\n\";\r\n // dequeue round\r\n startDeq.store(true);\r\n while (barrier.load() != numThreads) this_thread::yield();\r\n startDeq.store(false);\r\n tmp = numThreads;\r\n if (!barrier.compare_exchange_strong(tmp, 0)) cout << \"ERROR: CAS\\n\";\r\n }\r\n for (int tid = 0; tid < numThreads; tid++) burstThreads[tid].join();\r\n delete queue;\r\n }\r\n\r\n // Accounting\r\n vector agg(numRuns);\r\n for (int irun = 0; irun < numRuns; irun++) {\r\n nanoseconds maxNsEnq = 0ns;\r\n nanoseconds maxNsDeq = 0ns;\r\n for (int tid = 0; tid < numThreads; tid++) {\r\n if (results[tid][irun].nsEnq > maxNsEnq) maxNsEnq = results[tid][irun].nsEnq;\r\n if (results[tid][irun].nsDeq > maxNsDeq) maxNsDeq = results[tid][irun].nsDeq;\r\n agg[irun].numEnq += results[tid][irun].numEnq;\r\n agg[irun].numDeq += results[tid][irun].numDeq;\r\n }\r\n agg[irun].nsEnq = maxNsEnq;\r\n agg[irun].nsDeq = maxNsDeq;\r\n agg[irun].totOpsSec = agg[irun].nsEnq.count()+agg[irun].nsDeq.count();\r\n }\r\n\r\n // Compute the median. numRuns should be an odd number\r\n sort(agg.begin(),agg.end());\r\n Result median = agg[numRuns/2];\r\n const long long allThreadsEnqPerSec = median.numEnq*NSEC_IN_SEC/median.nsEnq.count();\r\n const long long allThreadsDeqPerSec = median.numDeq*NSEC_IN_SEC/median.nsDeq.count();\r\n\r\n // Printed value is the median of the number of ops per second that all threads were able to accomplish (on average)\r\n cout << \"Enq/sec = \" << allThreadsEnqPerSec << \" Deq/sec = \" << allThreadsDeqPerSec << \"\\n\";\r\n resultsEnq = allThreadsEnqPerSec;\r\n resultsDeq = allThreadsDeqPerSec;\r\n }\r\n};\r\n\r\n#endif\r\n" }, { "path": "graphs/BenchmarkSPS.hpp", "content": "/*\r\n * Copyright 2017-2018\r\n * Andreia Correia \r\n * Pedro Ramalhete \r\n * Pascal Felber \r\n * Nachshon Cohen \r\n *\r\n * This work is published under the MIT license. See LICENSE.txt\r\n */\r\n#ifndef _BENCHMARK_SPS_H_\r\n#define _BENCHMARK_SPS_H_\r\n\r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \r\n\r\nstatic const long arraySize = 1000; // 1k or 1M entries in the SPS array\r\n\r\n\r\nusing namespace std;\r\nusing namespace chrono;\r\n\r\n\r\n/**\r\n * This is a micro-benchmark\r\n */\r\nclass BenchmarkSPS {\r\n\r\nprivate:\r\n int numThreads;\r\n\r\npublic:\r\n struct UserData {\r\n long long seq;\r\n int tid;\r\n UserData(long long lseq, int ltid) {\r\n this->seq = lseq;\r\n this->tid = ltid;\r\n }\r\n UserData() {\r\n this->seq = -2;\r\n this->tid = -2;\r\n }\r\n UserData(const UserData &other) : seq(other.seq), tid(other.tid) { }\r\n\r\n bool operator < (const UserData& other) const {\r\n return seq < other.seq;\r\n }\r\n };\r\n\r\n BenchmarkSPS(int numThreads) {\r\n this->numThreads = numThreads;\r\n }\r\n\r\n\r\n /*\r\n * An array of integers that gets randomly permutated.\r\n */\r\n template class TMTYPE>\r\n uint64_t benchmarkSPSInteger(std::string& className, const seconds testLengthSeconds, const long numSwapsPerTx, const int numRuns) {\r\n long long ops[numThreads][numRuns];\r\n long long lengthSec[numRuns];\r\n atomic startFlag = { false };\r\n atomic quit = { false };\r\n\r\n className = TM::className();\r\n cout << \"##### \" << TM::className() << \" ##### \\n\";\r\n\r\n // Create the array of integers and initialize it\r\n TMTYPE* parray;\r\n // It's ok to capture by reference, we're running single-threaded now\r\n TM::updateTx([&] () {\r\n parray = new TMTYPE[arraySize];\r\n //parray = (TMTYPE*)TM::tmMalloc(sizeof(TMTYPE)*arraySize);\r\n } );\r\n // Break up the initialization into transactions of 1k stores, so it fits in the log\r\n for (long j = 0; j < arraySize; j+=1000) {\r\n TM::updateTx([&] () {\r\n for (int i = 0; i < 1000 && i+j < arraySize; i++) parray[i+j] = i+j;\r\n } );\r\n }\r\n\r\n auto func = [this,&startFlag,&quit,&numSwapsPerTx,&parray](long long *ops, const int tid) {\r\n uint64_t seed = tid+1234567890123456781ULL;\r\n // Spin until the startFlag is set\r\n while (!startFlag.load()) {}\r\n // Do transactions until the quit flag is set\r\n long long tcount = 0;\r\n while (!quit.load()) {\r\n TM::updateTx([&] () {\r\n for (int i = 0; i < numSwapsPerTx; i++) {\r\n seed = randomLong(seed);\r\n auto ia = seed%arraySize;\r\n uint64_t tmp = parray[ia];\r\n seed = randomLong(seed);\r\n auto ib = seed%arraySize;\r\n parray[ia] = parray[ib];\r\n parray[ib] = tmp;\r\n }\r\n } );\r\n ++tcount;\r\n /*\r\n PE::read_transaction([this,&seed,&parray,&numWordsPerTransaction] () {\r\n PersistentArrayInt* read_array = PE::template get_object>(PIDX_INT_ARRAY);\r\n // Check that the array is consistent\r\n int sum = 0;\r\n for (int i = 0; i < arraySize; i++) {\r\n sum += read_array->counters[i];\r\n }\r\n assert(sum == 0);\r\n } );\r\n */\r\n }\r\n *ops = tcount;\r\n };\r\n for (int irun = 0; irun < numRuns; irun++) {\r\n if (irun == 0) className = TM::className();\r\n thread enqdeqThreads[numThreads];\r\n for (int tid = 0; tid < numThreads; tid++) enqdeqThreads[tid] = thread(func, &ops[tid][irun], tid);\r\n auto startBeats = steady_clock::now();\r\n startFlag.store(true);\r\n // Sleep for 20 seconds\r\n this_thread::sleep_for(testLengthSeconds);\r\n quit.store(true);\r\n auto stopBeats = steady_clock::now();\r\n for (int tid = 0; tid < numThreads; tid++) enqdeqThreads[tid].join();\r\n lengthSec[irun] = (stopBeats-startBeats).count();\r\n startFlag.store(false);\r\n quit.store(false);\r\n }\r\n\r\n // It's ok to capture by reference, we're running single-threaded now\r\n TM::updateTx([&] () {\r\n delete[] parray;\r\n //TM::tmFree(parray);\r\n });\r\n\r\n // Accounting\r\n vector agg(numRuns);\r\n for (int irun = 0; irun < numRuns; irun++) {\r\n \tfor(int i=0;i class TMTYPE, typename TMBASE>\r\n uint64_t benchmarkSPSObject(std::string& className, const seconds testLengthSeconds, const long numSwapsPerTx, const int numRuns) {\r\n long long ops[numThreads][numRuns];\r\n long long lengthSec[numRuns];\r\n atomic startFlag = { false };\r\n atomic quit = { false };\r\n\r\n struct MyObject : public TMBASE {\r\n uint64_t a {0}; // For the OneFile STMs these don't need to be tmtypes because they're immutable after visible in this benchmark\r\n uint64_t b {0};\r\n MyObject(uint64_t a0, uint64_t b0) {\r\n a = a0;\r\n b = b0;\r\n }\r\n MyObject(const MyObject &other) {\r\n a = other.a;\r\n b = other.b;\r\n }\r\n };\r\n\r\n // Create the array of integers and initialize it\r\n TMTYPE* parray;\r\n parray = new TMTYPE[arraySize];\r\n // Break up the initialization into transactions of 1k stores, so it fits in the log\r\n for (long j = 0; j < arraySize; j+=1000) {\r\n TM::updateTx([&] () {\r\n for (int i = 0; i < 1000 && i+j < arraySize; i++) parray[i+j] = TM::template tmNew((uint64_t)i+j,(uint64_t)i);\r\n } );\r\n }\r\n /*\r\n TM::updateTx([&] () {\r\n for (int i = 0; i < arraySize; i++) parray[i] = TM::template tmNew((uint64_t)i,(uint64_t)i);\r\n } );\r\n */\r\n\r\n\r\n auto func = [this,&startFlag,&quit,&numSwapsPerTx,&parray](long long *ops, const int tid) {\r\n uint64_t seed = tid+1234567890123456781ULL;\r\n // Spin until the startFlag is set\r\n while (!startFlag.load()) {}\r\n // Do transactions until the quit flag is set\r\n long long tcount = 0;\r\n while (!quit.load()) {\r\n TM::updateTx([&] () {\r\n for (int i = 0; i < numSwapsPerTx; i++) {\r\n seed = randomLong(seed);\r\n auto ia = seed%arraySize;\r\n // Create a new object with the same contents to replace the old object, at a random location\r\n MyObject* tmp = TM::template tmNew(*parray[ia]);\r\n TM::template tmDelete(parray[ia]);\r\n parray[ia] = tmp;\r\n }\r\n } );\r\n ++tcount;\r\n }\r\n *ops = tcount;\r\n };\r\n for (int irun = 0; irun < numRuns; irun++) {\r\n if (irun == 0) {\r\n className = TM::className();\r\n cout << \"##### \" << TM::className() << \" ##### \\n\";\r\n }\r\n thread enqdeqThreads[numThreads];\r\n for (int tid = 0; tid < numThreads; tid++) enqdeqThreads[tid] = thread(func, &ops[tid][irun], tid);\r\n auto startBeats = steady_clock::now();\r\n startFlag.store(true);\r\n // Sleep for 20 seconds\r\n this_thread::sleep_for(testLengthSeconds);\r\n quit.store(true);\r\n auto stopBeats = steady_clock::now();\r\n for (int tid = 0; tid < numThreads; tid++) enqdeqThreads[tid].join();\r\n lengthSec[irun] = (stopBeats-startBeats).count();\r\n startFlag.store(false);\r\n quit.store(false);\r\n }\r\n TM::updateTx([&] () {\r\n for (int i = 0; i < arraySize; i++) TM::template tmDelete(parray[i]);\r\n });\r\n delete[] parray;\r\n\r\n // Accounting\r\n vector agg(numRuns);\r\n for (int irun = 0; irun < numRuns; irun++) {\r\n for(int i=0;i> 12; // a\r\n x ^= x << 25; // b\r\n x ^= x >> 27; // c\r\n return x * 2685821657736338717LL;\r\n }\r\n};\r\n\r\n#endif\r\n" }, { "path": "graphs/BenchmarkSets.hpp", "content": "/*\r\n * Copyright 2017-2018\r\n * Andreia Correia \r\n * Pedro Ramalhete \r\n * Pascal Felber \r\n * Nachshon Cohen \r\n *\r\n * This work is published under the MIT license. See LICENSE.txt\r\n */\r\n#ifndef _BENCHMARK_SETS_H_\r\n#define _BENCHMARK_SETS_H_\r\n\r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \r\n\r\nusing namespace std;\r\nusing namespace chrono;\r\n\r\n// Regular UserData\r\nstruct UserData {\r\n long long seq;\r\n int tid;\r\n UserData(long long lseq, int ltid=0) {\r\n this->seq = lseq;\r\n this->tid = ltid;\r\n }\r\n UserData() {\r\n this->seq = -2;\r\n this->tid = -2;\r\n }\r\n UserData(const UserData &other) : seq(other.seq), tid(other.tid) { }\r\n\r\n bool operator < (const UserData& other) const {\r\n return seq < other.seq;\r\n }\r\n bool operator == (const UserData& other) const {\r\n return seq == other.seq && tid == other.tid;\r\n }\r\n bool operator != (const UserData& other) const {\r\n return seq != other.seq || tid != other.tid;\r\n }\r\n};\r\n\r\n\r\nnamespace std {\r\n template <>\r\n struct hash {\r\n std::size_t operator()(const UserData& k) const {\r\n using std::size_t;\r\n using std::hash;\r\n return (hash()(k.seq)); // This hash has no collisions, which is irealistic\r\n }\r\n };\r\n}\r\n\r\n\r\n/**\r\n * This is a micro-benchmark of sets\r\n */\r\nclass BenchmarkSets {\r\n\r\nprivate:\r\n struct Result {\r\n nanoseconds nsEnq = 0ns;\r\n nanoseconds nsDeq = 0ns;\r\n long long numEnq = 0;\r\n long long numDeq = 0;\r\n long long totOpsSec = 0;\r\n\r\n Result() { }\r\n\r\n Result(const Result &other) {\r\n nsEnq = other.nsEnq;\r\n nsDeq = other.nsDeq;\r\n numEnq = other.numEnq;\r\n numDeq = other.numDeq;\r\n totOpsSec = other.totOpsSec;\r\n }\r\n\r\n bool operator < (const Result& other) const {\r\n return totOpsSec < other.totOpsSec;\r\n }\r\n };\r\n\r\n static const long long NSEC_IN_SEC = 1000000000LL;\r\n\r\n int numThreads;\r\n\r\npublic:\r\n BenchmarkSets(int numThreads) {\r\n this->numThreads = numThreads;\r\n }\r\n\r\n\r\n /**\r\n * When doing \"updates\" we execute a random removal and if the removal is successful we do an add() of the\r\n * same item immediately after. This keeps the size of the data structure equal to the original size (minus\r\n * MAX_THREADS items at most) which gives more deterministic results.\r\n */\r\n template\r\n long long benchmark(std::string& className, const int updateRatio, const seconds testLengthSeconds, const int numRuns, const int numElements, const bool dedicated=false) {\r\n long long ops[numThreads][numRuns];\r\n long long lengthSec[numRuns];\r\n atomic quit = { false };\r\n atomic startFlag = { false };\r\n\r\n className = S::className();\r\n std::cout << \"##### \" << S::className() << \" ##### \\n\";\r\n S* set = new S(numThreads);\r\n // Create all the keys in the concurrent set\r\n K** udarray = new K*[numElements];\r\n for (int i = 0; i < numElements; i++) udarray[i] = new K(i);\r\n // Add all the items to the list\r\n set->addAll(udarray, numElements, 0);\r\n\r\n // Can either be a Reader or a Writer\r\n auto rw_lambda = [this,&quit,&startFlag,&set,&udarray,&numElements](const int updateRatio, long long *ops, const int tid) {\r\n long long numOps = 0;\r\n while (!startFlag.load()) ; // spin\r\n uint64_t seed = tid+1234567890123456781ULL;\r\n while (!quit.load()) {\r\n seed = randomLong(seed);\r\n int update = seed%1000;\r\n seed = randomLong(seed);\r\n auto ix = (unsigned int)(seed%numElements);\r\n if (update < updateRatio) {\r\n // I'm a Writer\r\n if (set->remove(*udarray[ix], tid)) {\r\n \tnumOps++;\r\n \tset->add(*udarray[ix], tid);\r\n }\r\n numOps++;\r\n } else {\r\n \t// I'm a Reader\r\n set->contains(*udarray[ix], tid);\r\n seed = randomLong(seed);\r\n ix = (unsigned int)(seed%numElements);\r\n set->contains(*udarray[ix], tid);\r\n numOps += 2;\r\n }\r\n }\r\n *ops = numOps;\r\n };\r\n\r\n for (int irun = 0; irun < numRuns; irun++) {\r\n thread rwThreads[numThreads];\r\n if (dedicated) {\r\n rwThreads[0] = thread(rw_lambda, 1000, &ops[0][irun], 0);\r\n rwThreads[1] = thread(rw_lambda, 1000, &ops[1][irun], 1);\r\n for (int tid = 2; tid < numThreads; tid++) rwThreads[tid] = thread(rw_lambda, updateRatio, &ops[tid][irun], tid);\r\n } else {\r\n for (int tid = 0; tid < numThreads; tid++) rwThreads[tid] = thread(rw_lambda, updateRatio, &ops[tid][irun], tid);\r\n }\r\n this_thread::sleep_for(100ms);\r\n auto startBeats = steady_clock::now();\r\n startFlag.store(true);\r\n // Sleep for testLengthSeconds seconds\r\n this_thread::sleep_for(testLengthSeconds);\r\n quit.store(true);\r\n auto stopBeats = steady_clock::now();\r\n for (int tid = 0; tid < numThreads; tid++) rwThreads[tid].join();\r\n lengthSec[irun] = (stopBeats-startBeats).count();\r\n if (dedicated) {\r\n // We don't account for the write-only operations but we aggregate the values from the two threads and display them\r\n std::cout << \"Mutative transactions per second = \" << (ops[0][irun] + ops[1][irun])*1000000000LL/lengthSec[irun] << \"\\n\";\r\n ops[0][irun] = 0;\r\n ops[1][irun] = 0;\r\n }\r\n quit.store(false);\r\n startFlag.store(false);\r\n // Compute ops at the end of each run\r\n long long agg = 0;\r\n for (int tid = 0; tid < numThreads; tid++) {\r\n agg += ops[tid][irun]*1000000000LL/lengthSec[irun];\r\n }\r\n }\r\n\r\n // Clear the set, one key at a time and then delete the instance\r\n for (int i = 0; i < numElements; i++) set->remove(*udarray[i], 0);\r\n delete set;\r\n\r\n for (int i = 0; i < numElements; i++) delete udarray[i];\r\n delete[] udarray;\r\n\r\n // Accounting\r\n vector agg(numRuns);\r\n for (int irun = 0; irun < numRuns; irun++) {\r\n for (int tid = 0; tid < numThreads; tid++) {\r\n agg[irun] += ops[tid][irun]*1000000000LL/lengthSec[irun];\r\n }\r\n }\r\n\r\n // Compute the median. numRuns must be an odd number\r\n sort(agg.begin(),agg.end());\r\n auto maxops = agg[numRuns-1];\r\n auto minops = agg[0];\r\n auto medianops = agg[numRuns/2];\r\n auto delta = (long)(100.*(maxops-minops) / ((double)medianops));\r\n // Printed value is the median of the number of ops per second that all threads were able to accomplish (on average)\r\n std::cout << \"Ops/sec = \" << medianops << \" delta = \" << delta << \"% min = \" << minops << \" max = \" << maxops << \"\\n\";\r\n return medianops;\r\n }\r\n\r\n\r\n\r\n /*\r\n * Inspired by Trevor Brown's benchmarks (does everyone else do it like this?)\r\n */\r\n template\r\n long long benchmarkRandomFill(std::string& className, const int updateRatio, const seconds testLengthSeconds, const int numRuns, const int numElements, const bool dedicated=false) {\r\n long long ops[numThreads][numRuns];\r\n long long lengthSec[numRuns];\r\n atomic quit = { false };\r\n atomic startFlag = { false };\r\n\r\n className = S::className();\r\n std::cout << \"##### \" << S::className() << \" ##### \\n\";\r\n S* set = new S(numThreads);\r\n // Create all the keys in the concurrent set\r\n K** udarray = new K*[2*numElements];\r\n for (int i = 0; i < 2*numElements; i++) udarray[i] = new K(i);\r\n // Add half the keys to the list\r\n long ielem = 0;\r\n uint64_t seed = 1234567890123456781ULL;\r\n while (ielem < numElements/2) {\r\n seed = randomLong(seed);\r\n // Insert new random keys until we have 'numElements/2' keys in the tree\r\n if (set->add(*udarray[seed%(numElements)], 0)) ielem++;\r\n }\r\n // Add all keys, repeating if needed\r\n set->addAll(udarray, numElements, 0);\r\n\r\n // Can either be a Reader or a Writer\r\n auto rw_lambda = [this,&quit,&startFlag,&set,&udarray,&numElements](const int updateRatio, long long *ops, const int tid) {\r\n long long numOps = 0;\r\n while (!startFlag.load()) ; // spin\r\n uint64_t seed = tid+1234567890123456781ULL;\r\n while (!quit.load()) {\r\n seed = randomLong(seed);\r\n int update = seed%1000;\r\n seed = randomLong(seed);\r\n auto ix = (unsigned int)(seed%numElements);\r\n if (update < updateRatio) {\r\n // I'm a Writer\r\n if (set->remove(*udarray[ix], tid)) {\r\n numOps++;\r\n set->add(*udarray[ix], tid);\r\n }\r\n numOps++;\r\n } else {\r\n // I'm a Reader\r\n set->contains(*udarray[ix], tid);\r\n seed = randomLong(seed);\r\n ix = (unsigned int)(seed%numElements);\r\n set->contains(*udarray[ix], tid);\r\n numOps += 2;\r\n }\r\n }\r\n *ops = numOps;\r\n };\r\n\r\n for (int irun = 0; irun < numRuns; irun++) {\r\n thread rwThreads[numThreads];\r\n if (dedicated) {\r\n rwThreads[0] = thread(rw_lambda, 1000, &ops[0][irun], 0);\r\n rwThreads[1] = thread(rw_lambda, 1000, &ops[1][irun], 1);\r\n for (int tid = 2; tid < numThreads; tid++) rwThreads[tid] = thread(rw_lambda, updateRatio, &ops[tid][irun], tid);\r\n } else {\r\n for (int tid = 0; tid < numThreads; tid++) rwThreads[tid] = thread(rw_lambda, updateRatio, &ops[tid][irun], tid);\r\n }\r\n this_thread::sleep_for(100ms);\r\n auto startBeats = steady_clock::now();\r\n startFlag.store(true);\r\n // Sleep for testLengthSeconds seconds\r\n this_thread::sleep_for(testLengthSeconds);\r\n quit.store(true);\r\n auto stopBeats = steady_clock::now();\r\n for (int tid = 0; tid < numThreads; tid++) rwThreads[tid].join();\r\n lengthSec[irun] = (stopBeats-startBeats).count();\r\n if (dedicated) {\r\n // We don't account for the write-only operations but we aggregate the values from the two threads and display them\r\n std::cout << \"Mutative transactions per second = \" << (ops[0][irun] + ops[1][irun])*1000000000LL/lengthSec[irun] << \"\\n\";\r\n ops[0][irun] = 0;\r\n ops[1][irun] = 0;\r\n }\r\n quit.store(false);\r\n startFlag.store(false);\r\n // Compute ops at the end of each run\r\n long long agg = 0;\r\n for (int tid = 0; tid < numThreads; tid++) {\r\n agg += ops[tid][irun]*1000000000LL/lengthSec[irun];\r\n }\r\n }\r\n\r\n /* Clear the tree, one key at a time and then delete the instance */\r\n for (int i = 0; i < numElements; i++) set->remove(*udarray[i], 0);\r\n delete set;\r\n\r\n for (int i = 0; i < numElements; i++) delete udarray[i];\r\n delete[] udarray;\r\n\r\n // Accounting\r\n vector agg(numRuns);\r\n for (int irun = 0; irun < numRuns; irun++) {\r\n for (int tid = 0; tid < numThreads; tid++) {\r\n agg[irun] += ops[tid][irun]*1000000000LL/lengthSec[irun];\r\n }\r\n }\r\n\r\n // Compute the median. numRuns must be an odd number\r\n sort(agg.begin(),agg.end());\r\n auto maxops = agg[numRuns-1];\r\n auto minops = agg[0];\r\n auto medianops = agg[numRuns/2];\r\n auto delta = (long)(100.*(maxops-minops) / ((double)medianops));\r\n // Printed value is the median of the number of ops per second that all threads were able to accomplish (on average)\r\n std::cout << \"Ops/sec = \" << medianops << \" delta = \" << delta << \"% min = \" << minops << \" max = \" << maxops << \"\\n\";\r\n return medianops;\r\n }\r\n\r\n /**\r\n * An imprecise but fast random number generator\r\n */\r\n uint64_t randomLong(uint64_t x) {\r\n x ^= x >> 12; // a\r\n x ^= x << 25; // b\r\n x ^= x >> 27; // c\r\n return x * 2685821657736338717LL;\r\n }\r\n};\r\n\r\n#endif\r\n" }, { "path": "graphs/Makefile", "content": "CXX = g++-8\nCXXFLAGS = -std=c++17 -g -O2 -DPWB_IS_CLFLUSHOPT # -fuse-ld=gold -fsanitize=address\n# For castor-1\n#CXXFLAGS = -std=c++17 -g -O2 -DPWB_IS_CLWB -DPM_REGION_SIZE=64*1024*1024*1024ULL -DPM_USE_DAX -DPM_FILE_NAME=\"\\\"/mnt/pmem0/durable\\\"\"\n\n# Possible options for PWB are:\n# -DPWB_IS_CLFLUSH\t\tpwb is a CLFLUSH and pfence/psync are nops (Broadwell)\n# -DPWB_IS_CLFLUSHOPT\tpwb is a CLFLUSHOPT and pfence/psync are SFENCE (Kaby Lake) \n# -DPWB_IS_CLWB\t\t\tpwb is a CLWB and pfence/psync are SFENCE (Sky Lake SP, or Canon Lake SP and beyond)\n# -DPWB_IS_NOP\t\t\tpwb/pfence/psync are nops. Used for shared memory persistence\n\n\nINCLUDES = -I../ -I../common/ \n#LIBS = -l/home/vagrant/tinystm/lib/libstm.a\n\n# This library is needed for ESTM\nESTM_LIB = -L../stms/estm-0.3.0/lib/ -lstm -lpthread\n\n# This library is needed for TinySTM\nTINYSTM_LIB = -L../stms/tinystm/lib/ -lstm -lpthread\nTINYSTM_INC = -I../stms/tinystm/\n\n# This library is needed for PMDK\nPMDKLIBS = -L/usr/local/lib -lpmemobj\n\n\nBINARIES = \\\n\tbin/sps-integer \\\n\tbin/sps-integer-tiny \\\n\tbin/sps-object \\\n\tbin/sps-object-tiny \\\n\tbin/set-ll-1k \\\n\tbin/set-ll-1k-tiny \\\n\tbin/set-ll-10k \\\n\tbin/set-ll-10k-tiny \\\n\tbin/set-tree-1k \\\n\tbin/set-tree-1k-tiny \\\n\tbin/set-tree-10k \\\n\tbin/set-tree-10k-tiny \\\n\tbin/set-tree-1m \\\n\tbin/set-tree-1m-tiny \\\n\tbin/set-hash-1k \\\n\tbin/set-hash-1k-tiny \\\n\tbin/q-ll-enq-deq \\\n\tbin/q-ll-enq-deq-tiny \\\n\tbin/q-array-enq-deq \\\n\tbin/q-array-enq-deq-tiny \\\n\tbin/psps-integer \\\n\tbin/pset-ll-1k \\\n\tbin/pset-ll-10k \\\n\tbin/pset-hash-1k \\\n\tbin/pset-tree-1k \\\n\tbin/pq-ll-enq-deq \\\n\tbin/latency-counter \\\n\tbin/latency-counter-tiny \\\n bin/pset-tree-1m-oflf \\\n bin/pset-tree-1m-ofwf \\\n bin/pset-tree-1m-pmdk \\\n\tbin/pset-tree-1m-romlog \\\n\tbin/pset-tree-1m-romlr \\\n#\tbin/pset-tree-1m-pmdk \\\n#\tbin/pread-while-writing-romlog \\\n\tbin/pread-while-writing-romlr \\\n\tbin/pread-while-writing-oflf \\\n\tbin/pread-while-writing-ofwf \\\n\tbin/pread-while-writing-pmdk \\\n\nSTMS = \\\n\t../stms/CRWWPSTM.hpp \\\n\t../stms/OneFileLF.hpp \\\n\t../stms/OneFileWF.hpp \\\n\t../stms/TinySTM.hpp \\\n\t../stms/tinystm/lib/libstm.a \\\n\nPTMS = \\\n\t../ptms/OneFilePTMLF.hpp \\\n\t../ptms/OneFilePTMWF.hpp \\\n\t../ptms/PMDKTM.hpp \\\n\tlib/libromulus.a \\\n ../ptms/romuluslog/RomulusLog.hpp \\\n ../ptms/romuluslr/RomulusLR.hpp \\\n\nSRC_LISTS = \\\n\t../datastructures/linkedlists/CRWWPLinkedListSet.hpp \\\n\t../datastructures/linkedlists/ESTMLinkedListSet.hpp \\\n\t../datastructures/linkedlists/OFLFLinkedListSet.hpp \\\n\t../datastructures/linkedlists/OFWFLinkedListSet.hpp \\\n\t../datastructures/linkedlists/STMLinkedListSet.hpp \\\n\t../datastructures/linkedlists/TinySTMLinkedListSet.hpp \\\n\t\nSRC_TREES = \\\n\t../datastructures/treemaps/ESTMRedBlackTree.hpp \\\n\t../datastructures/treemaps/NatarajanTreeHE.hpp \\\n\t../datastructures/treemaps/OFLFRedBlackTree.hpp \\\n\t../datastructures/treemaps/OFWFRedBlackTree.hpp \\\n\nQUEUES_DEP = \\\n\t../datastructures/queues/ESTMArrayLinkedListQueue.hpp \\\n\t../datastructures/queues/ESTMLinkedListQueue.hpp \\\n\t../datastructures/queues/FAAArrayQueue.hpp \\\n\t../datastructures/queues/LCRQueue.hpp \\\n\t../datastructures/queues/MichaelScottQueue.hpp \\\n\t../datastructures/queues/OFLFArrayLinkedListQueue.hpp \\\n\t../datastructures/queues/OFLFLinkedListQueue.hpp \\\n\t../datastructures/queues/OFWFArrayLinkedListQueue.hpp \\\n\t../datastructures/queues/OFWFLinkedListQueue.hpp \\\n\t../datastructures/queues/TurnQueue.hpp \\\n\t\nPQUEUES_DEP = \\\n\t../pdatastructures/pqueues/MichaelScottQueue.hpp \\\n\t../pdatastructures/pqueues/PFriedmanQueue.hpp \\\n\t../pdatastructures/pqueues/PMDKLinkedListQueue.hpp \\\n\t../pdatastructures/pqueues/PMichaelScottQueue.hpp \\\n\t../pdatastructures/TMLinkedListQueue.hpp \\\n\t../pdatastructures/pqueues/POFLFLinkedListQueue.hpp \\\n\t../pdatastructures/pqueues/POFWFLinkedListQueue.hpp \\\n\t../pdatastructures/pqueues/RomLogLinkedListQueue.hpp \\\n\t../pdatastructures/pqueues/RomLRLinkedListQueue.hpp \\\n\nROMULUS_LIB_SRC = \\\n\t../common/ThreadRegistry.cpp \\\n\t../ptms/romuluslog/malloc.cpp \\\n\t../ptms/romuluslog/RomulusLog.cpp \\\n\t../ptms/romuluslr/malloc.cpp \\\n\t../ptms/romuluslr/RomulusLR.cpp \\\n\nROMULUS_LIB_DEP = \\\n\t$(ROMULUS_LIB_SRC) \\\n\t../ptms/romuluslog/RomulusLog.hpp \\\n\t../ptms/romuluslr/RomulusLR.hpp \\\n\t\t\nTREVOR_BROWN_INCLUDES = \\\n\t-I../datastructures/trevor_brown_abtree/common/recordmgr \\\n\t-I../datastructures/trevor_brown_abtree/common \\\n\t-I../datastructures/trevor_brown_abtree/common/descriptors \\\n\t-I../datastructures/trevor_brown_abtree/common/rq \\\n\t-I../datastructures/trevor_brown_abtree/common/rq \\\n\t-I../datastructures/trevor_brown_abtree/common/atomic_ops \\\n\nall: $(BINARIES) persistencyclean\n\n\nclean: persistencyclean\n\trm -f bin/*\n\trm -f lib/*\n\n\npersistencyclean:\n\trm -f /dev/shm/*_shared\n\trm -f /dev/shm/psegments/*\n\n\n#\n# Create a library for RomulusLog and RomulusLR\n#\nlib/threadregistry.o: $(ROMULUS_LIB_DEP)\n\t$(CXX) $(CXXFLAGS) $(INCLUDES) -c ../common/ThreadRegistry.cpp -o lib/threadregistry.o\n\nlib/mallocromlog.o: $(ROMULUS_LIB_DEP)\n\t$(CXX) $(CXXFLAGS) $(INCLUDES) -c ../ptms/romuluslog/malloc.cpp -o lib/mallocromlog.o\n\nlib/romlog.o: $(ROMULUS_LIB_DEP)\n\t$(CXX) $(CXXFLAGS) $(INCLUDES) -c ../ptms/romuluslog/RomulusLog.cpp -o lib/romlog.o\n\nlib/mallocromlr.o: $(ROMULUS_LIB_DEP)\n\t$(CXX) $(CXXFLAGS) $(INCLUDES) -c ../ptms/romuluslr/malloc.cpp -o lib/mallocromlr.o\n\nlib/romlr.o: $(ROMULUS_LIB_DEP)\n\t$(CXX) $(CXXFLAGS) $(INCLUDES) -c ../ptms/romuluslr/RomulusLR.cpp -o lib/romlr.o\n\t\nlib/libromulus.a: lib/threadregistry.o lib/mallocromlog.o lib/romlog.o lib/mallocromlr.o lib/romlr.o\n\tar rcs lib/libromulus.a lib/threadregistry.o lib/mallocromlog.o lib/romlog.o lib/mallocromlr.o lib/romlr.o\n\n\n\n#\n# Queues for volatile memory\n#\t\nbin/q-ll-enq-deq: q-ll-enq-deq.cpp $(STMS) $(QUEUES_DEP)\n\t$(CXX) $(CXXFLAGS) $(INCLUDES) $(CSRCS) q-ll-enq-deq.cpp -o bin/q-ll-enq-deq -lpthread $(ESTM_LIB)\n\t\nbin/q-array-enq-deq: q-array-enq-deq.cpp $(STMS) $(QUEUES_DEP)\n\t$(CXX) $(CXXFLAGS) $(INCLUDES) $(CSRCS) q-array-enq-deq.cpp -o bin/q-array-enq-deq -lpthread $(ESTM_LIB)\n\t\nbin/q-ll-burst: q-ll-burst.cpp $(QUEUES_DEP)\n\t$(CXX) $(CXXFLAGS) $(INCLUDES) $(CSRCS) q-ll-burst.cpp -o bin/q-ll-burst -lpthread $(ESTM_LIB)\n\n# Same as above but for TinySTM\nbin/q-ll-enq-deq-tiny: q-ll-enq-deq.cpp $(STMS) $(QUEUES_DEP)\n\t$(CXX) $(CXXFLAGS) -DUSE_TINY $(INCLUDES) $(TINYSTM_INC) $(CSRCS) q-ll-enq-deq.cpp -o bin/q-ll-enq-deq-tiny -lpthread $(TINYSTM_LIB)\n\nbin/q-array-enq-deq-tiny: q-array-enq-deq.cpp $(STMS) $(QUEUES_DEP)\n\t$(CXX) $(CXXFLAGS) -DUSE_TINY $(INCLUDES) $(TINYSTM_INC) $(CSRCS) q-array-enq-deq.cpp -o bin/q-array-enq-deq-tiny -lpthread $(TINYSTM_LIB)\n\n\n#\n# Queues for persistent memory\n#\t\nbin/pq-ll-enq-deq: pq-ll-enq-deq.cpp $(PTMS) $(PQUEUES_DEP) PBenchmarkQueues.hpp\n\t$(CXX) $(CXXFLAGS) $(INCLUDES) pq-ll-enq-deq.cpp -o bin/pq-ll-enq-deq -lpthread $(PMDKLIBS) lib/libromulus.a\n\n\t\n#\n# Sets for volatile memory\n#\t\nbin/set-ll-1k: set-ll-1k.cpp $(STMS) $(SRC_LISTS)\n\t$(CXX) $(CXXFLAGS) $(INCLUDES) $(CSRCS) set-ll-1k.cpp -o bin/set-ll-1k -lpthread $(ESTM_LIB)\n\nbin/set-ll-10k: set-ll-10k.cpp $(STMS) $(SRC_LISTS)\n\t$(CXX) $(CXXFLAGS) $(INCLUDES) $(CSRCS) set-ll-10k.cpp -o bin/set-ll-10k -lpthread $(ESTM_LIB)\n\nbin/set-tree-1k: set-tree-1k.cpp $(STMS) $(SRC_TREES)\n\t$(CXX) $(CXXFLAGS) -fuse-ld=gold -fsanitize=address $(INCLUDES) $(TREVOR_BROWN_INCLUDES) ../common/ThreadRegistry.cpp $(CSRCS) set-tree-1k.cpp -o bin/set-tree-1k -lpthread $(ESTM_LIB)\n\nbin/set-tree-10k: set-tree-10k.cpp $(STMS) $(SRC_TREES)\n\t$(CXX) $(CXXFLAGS) $(INCLUDES) $(TREVOR_BROWN_INCLUDES) ../common/ThreadRegistry.cpp $(CSRCS) set-tree-10k.cpp -o bin/set-tree-10k -lpthread $(ESTM_LIB)\n\nbin/set-tree-1m: set-tree-1m.cpp $(STMS) $(SRC_TREES)\n\t$(CXX) $(CXXFLAGS) $(INCLUDES) $(TREVOR_BROWN_INCLUDES) ../common/ThreadRegistry.cpp $(CSRCS) set-tree-1m.cpp -o bin/set-tree-1m -lpthread $(ESTM_LIB)\n\nbin/set-hash-1k: set-hash-1k.cpp $(STMS)\n\t$(CXX) $(CXXFLAGS) $(INCLUDES) $(CSRCS) set-hash-1k.cpp -o bin/set-hash-1k -lpthread $(ESTM_LIB)\n\n\n\n# Same as above, but for Tiny STM only\nbin/set-ll-1k-tiny: set-ll-1k.cpp $(STMS) $(SRC_LISTS)\n\t$(CXX) $(CXXFLAGS) -DUSE_TINY $(INCLUDES) $(TINYSTM_INC) $(CSRCS) set-ll-1k.cpp -o bin/set-ll-1k-tiny -lpthread $(TINYSTM_LIB)\n\nbin/set-ll-10k-tiny: set-ll-10k.cpp $(STMS) $(SRC_LISTS)\n\t$(CXX) $(CXXFLAGS) -DUSE_TINY $(INCLUDES) $(TINYSTM_INC) $(CSRCS) set-ll-10k.cpp -o bin/set-ll-10k-tiny -lpthread $(TINYSTM_LIB)\n\nbin/set-tree-1k-tiny: set-tree-1k.cpp $(STMS)\n\t$(CXX) $(CXXFLAGS) -DUSE_TINY $(INCLUDES) $(TINYSTM_INC) $(CSRCS) set-tree-1k.cpp -o bin/set-tree-1k-tiny -lpthread $(TINYSTM_LIB)\n\nbin/set-tree-10k-tiny: set-tree-10k.cpp $(STMS)\n\t$(CXX) $(CXXFLAGS) -DUSE_TINY $(INCLUDES) $(TINYSTM_INC) $(CSRCS) set-tree-10k.cpp -o bin/set-tree-10k-tiny -lpthread $(TINYSTM_LIB)\n\nbin/set-tree-1m-tiny: set-tree-1m.cpp $(STMS)\n\t$(CXX) $(CXXFLAGS) -DUSE_TINY $(INCLUDES) $(TINYSTM_INC) $(CSRCS) set-tree-1m.cpp -o bin/set-tree-1m-tiny -lpthread $(TINYSTM_LIB)\n\nbin/set-hash-1k-tiny: set-hash-1k.cpp $(STMS)\n\t$(CXX) $(CXXFLAGS) -DUSE_TINY $(INCLUDES) $(TINYSTM_INC) $(CSRCS) set-hash-1k.cpp -o bin/set-hash-1k-tiny -lpthread $(TINYSTM_LIB)\n\n\n\n#\n# Sets for persistent memory\n#\t\nbin/pset-ll-1k: pset-ll-1k.cpp $(PTMS) PBenchmarkSets.hpp ../pdatastructures/TMLinkedListSet.hpp lib/libromulus.a\n\t$(CXX) $(CXXFLAGS) $(INCLUDES) pset-ll-1k.cpp -o bin/pset-ll-1k -lpthread $(PMDKLIBS) lib/libromulus.a\n\nbin/pset-ll-10k: pset-ll-10k.cpp $(PTMS) PBenchmarkSets.hpp ../pdatastructures/TMLinkedListSet.hpp lib/libromulus.a\n\t$(CXX) $(CXXFLAGS) $(INCLUDES) pset-ll-10k.cpp -o bin/pset-ll-10k -lpthread $(PMDKLIBS) lib/libromulus.a\n\nbin/pset-hash-1k: pset-hash-1k.cpp $(PTMS) PBenchmarkSets.hpp ../pdatastructures/TMHashMap.hpp lib/libromulus.a\n\t$(CXX) $(CXXFLAGS) $(INCLUDES) pset-hash-1k.cpp -o bin/pset-hash-1k -lpthread $(PMDKLIBS) lib/libromulus.a\n\nbin/pset-tree-1k: pset-tree-1k.cpp $(PTMS) PBenchmarkSets.hpp ../pdatastructures/TMRedBlackTree.hpp lib/libromulus.a\n\t$(CXX) $(CXXFLAGS) $(INCLUDES) pset-tree-1k.cpp -o bin/pset-tree-1k -lpthread $(PMDKLIBS) lib/libromulus.a\n\t\nbin/pset-tree-1m: pset-tree-1m.cpp $(PTMS) PBenchmarkSets.hpp ../pdatastructures/TMRedBlackTree.hpp lib/libromulus.a\n\t$(CXX) $(CXXFLAGS) $(INCLUDES) pset-tree-1m.cpp -o bin/pset-tree-1m -lpthread $(PMDKLIBS) lib/libromulus.a\n\n\t\n#\t\n# SPS for volatile memory\n#\nbin/sps-integer: sps-integer.cpp $(STMS) BenchmarkSPS.hpp\n\t$(CXX) $(CXXFLAGS) $(INCLUDES) $(CSRCS) sps-integer.cpp -o bin/sps-integer -lpthread $(ESTM_LIB)\n\nbin/sps-integer-tiny: sps-integer.cpp $(STMS) BenchmarkSPS.hpp\n\t$(CXX) $(CXXFLAGS) -DUSE_TINY $(INCLUDES) $(TINYSTM_INC) $(CSRCS) sps-integer.cpp -o bin/sps-integer-tiny -lpthread $(TINYSTM_LIB)\n\nbin/sps-object: sps-object.cpp $(STMS) BenchmarkSPS.hpp\n\t$(CXX) $(CXXFLAGS) $(INCLUDES) $(CSRCS) sps-object.cpp -o bin/sps-object -lpthread $(ESTM_LIB)\n\nbin/sps-object-tiny: sps-object.cpp $(STMS) BenchmarkSPS.hpp\n\t$(CXX) $(CXXFLAGS) -DUSE_TINY $(INCLUDES) $(TINYSTM_INC) $(CSRCS) sps-object.cpp -o bin/sps-object-tiny -lpthread $(TINYSTM_LIB)\n\n\n#\n# SPS for persistent memory\n#\nbin/psps-integer: psps-integer.cpp $(PTMS) PBenchmarkSPS.hpp lib/libromulus.a\n\t$(CXX) $(CXXFLAGS) $(INCLUDES) psps-integer.cpp -o bin/psps-integer -lpthread $(PMDKLIBS) lib/libromulus.a\n\nbin/psps-integer-atlas: psps-integer-atlas.cpp $(PTMS) PBenchmarkSPS.hpp persistencyclean lib/libromulus.a\n\t$(CXX) $(CXXFLAGS) $(INCLUDES) psps-integer-atlas.cpp -o bin/psps-integer-atlas -lpthread lib/libromulus.a\n\t\n\n# TODO: is it worth doing sps-object for PTMs ?\n\n\n#\n# Latency for STMs\n#\nbin/latency-counter: latency-counter.cpp $(STMS) BenchmarkLatencyCounter.hpp\n\t$(CXX) $(CXXFLAGS) $(INCLUDES) $(CSRCS) latency-counter.cpp -o bin/latency-counter -lpthread $(ESTM_LIB)\n\nbin/latency-counter-tiny: latency-counter.cpp $(STMS) BenchmarkLatencyCounter.hpp\n\t$(CXX) $(CXXFLAGS) -DUSE_TINY $(INCLUDES) $(TINYSTM_INC) $(CSRCS) latency-counter.cpp -o bin/latency-counter-tiny -lpthread $(TINYSTM_LIB)\n\n\n\n\t\n# \n# Persistent balanced tree with 1M keys. Must be compiled one at a time otherwise you get all the NVM heaps allocated, which is too much \n#\nbin/pset-tree-1m-romlog: pset-tree-1m.cpp PBenchmarkSets.hpp ../pdatastructures/TMRedBlackTree.hpp lib/libromulus.a \n\t$(CXX) $(CXXFLAGS) -DUSE_ROMLOG $(INCLUDES) pset-tree-1m.cpp -o bin/pset-tree-1m-romlog -lpthread lib/libromulus.a\n\nbin/pset-tree-1m-romlr: pset-tree-1m.cpp PBenchmarkSets.hpp ../pdatastructures/TMRedBlackTree.hpp lib/libromulus.a \n\t$(CXX) $(CXXFLAGS) -DUSE_ROMLR $(INCLUDES) pset-tree-1m.cpp -o bin/pset-tree-1m-romlr -lpthread lib/libromulus.a\n\nbin/pset-tree-1m-oflf: pset-tree-1m.cpp PBenchmarkSets.hpp ../pdatastructures/TMRedBlackTree.hpp ../ptms/OneFilePTMLF.hpp\n\t$(CXX) $(CXXFLAGS) -DUSE_OFLF $(INCLUDES) pset-tree-1m.cpp -o bin/pset-tree-1m-oflf -lpthread\n\nbin/pset-tree-1m-ofwf: pset-tree-1m.cpp PBenchmarkSets.hpp ../pdatastructures/TMRedBlackTree.hpp ../ptms/OneFilePTMWF.hpp\n\t$(CXX) $(CXXFLAGS) -DUSE_OFWF $(INCLUDES) pset-tree-1m.cpp -o bin/pset-tree-1m-ofwf -lpthread\n\nbin/pset-tree-1m-pmdk: pset-tree-1m.cpp PBenchmarkSets.hpp ../pdatastructures/TMRedBlackTree.hpp \n\t$(CXX) $(CXXFLAGS) -DUSE_PMDK $(INCLUDES) pset-tree-1m.cpp -o bin/pset-tree-1m-pmdk -lpthread $(PMDKLIBS)\n\t\n\n# experimental...\nbin/pread-while-writing-romlog: pread-while-writing.cpp PBenchmarkSets.hpp ../pdatastructures/TMRedBlackTree.hpp lib/libromulus.a \n\t$(CXX) $(CXXFLAGS) -DUSE_ROMLOG $(INCLUDES) pread-while-writing.cpp -o bin/pread-while-writing-romlog -lpthread lib/libromulus.a\n\nbin/pread-while-writing-romlr: pread-while-writing.cpp PBenchmarkSets.hpp ../pdatastructures/TMRedBlackTree.hpp lib/libromulus.a \n\t$(CXX) $(CXXFLAGS) -DUSE_ROMLR $(INCLUDES) pread-while-writing.cpp -o bin/pread-while-writing-romlr -lpthread lib/libromulus.a\n\nbin/pread-while-writing-oflf: pread-while-writing.cpp PBenchmarkSets.hpp ../pdatastructures/TMRedBlackTree.hpp ../ptms/OneFilePTMLF.hpp\n\t$(CXX) $(CXXFLAGS) -DUSE_OFLF $(INCLUDES) pread-while-writing.cpp -o bin/pread-while-writing-oflf -lpthread\n\nbin/pread-while-writing-ofwf: pread-while-writing.cpp PBenchmarkSets.hpp ../pdatastructures/TMRedBlackTree.hpp ../ptms/OneFilePTMWF.hpp\n\t$(CXX) $(CXXFLAGS) -DUSE_OFWF $(INCLUDES) pread-while-writing.cpp -o bin/pread-while-writing-ofwf -lpthread\n\nbin/pread-while-writing-pmdk: pread-while-writing.cpp PBenchmarkSets.hpp ../pdatastructures/TMRedBlackTree.hpp \n\t$(CXX) $(CXXFLAGS) -DUSE_PMDK $(INCLUDES) pread-while-writing.cpp -o bin/pread-while-writing-pmdk -lpthread $(PMDKLIBS)\n\t\n" }, { "path": "graphs/PBenchmarkQueues.hpp", "content": "/*\r\n * Copyright 2017-2018\r\n * Andreia Correia \r\n * Pedro Ramalhete \r\n * Pascal Felber \r\n * Nachshon Cohen \r\n *\r\n * This work is published under the MIT license. See LICENSE.txt\r\n */\r\n#ifndef _PERSISTENT_BENCHMARK_Q_H_\r\n#define _PERSISTENT_BENCHMARK_Q_H_\r\n\r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \r\n\r\n\r\nusing namespace std;\r\nusing namespace chrono;\r\n\r\nstruct UserData {\r\n long long seq;\r\n int tid;\r\n UserData(long long lseq, int ltid) {\r\n this->seq = lseq;\r\n this->tid = ltid;\r\n }\r\n UserData() {\r\n this->seq = -2;\r\n this->tid = -2;\r\n }\r\n UserData(const UserData &other) : seq(other.seq), tid(other.tid) { }\r\n\r\n bool operator < (const UserData& other) const {\r\n return seq < other.seq;\r\n }\r\n};\r\n\r\n\r\n/**\r\n * This is a micro-benchmark for persistent queues\r\n */\r\nclass PBenchmarkQueues {\r\n\r\nprivate:\r\n\r\n struct Result {\r\n nanoseconds nsEnq = 0ns;\r\n nanoseconds nsDeq = 0ns;\r\n long long numEnq = 0;\r\n long long numDeq = 0;\r\n long long totOpsSec = 0;\r\n\r\n Result() { }\r\n\r\n Result(const Result &other) {\r\n nsEnq = other.nsEnq;\r\n nsDeq = other.nsDeq;\r\n numEnq = other.numEnq;\r\n numDeq = other.numDeq;\r\n totOpsSec = other.totOpsSec;\r\n }\r\n\r\n bool operator < (const Result& other) const {\r\n return totOpsSec < other.totOpsSec;\r\n }\r\n };\r\n\r\n // Performance benchmark constants\r\n static const long long kNumPairsWarmup = 1000000LL; // Each threads does 1M iterations as warmup\r\n\r\n // Contants for Ping-Pong performance benchmark\r\n static const int kPingPongBatch = 1000; // Each thread starts by injecting 1k items in the queue\r\n\r\n\r\n static const long long NSEC_IN_SEC = 1000000000LL;\r\n\r\n int numThreads;\r\n\r\npublic:\r\n\r\n PBenchmarkQueues(int numThreads) {\r\n this->numThreads = numThreads;\r\n }\r\n\r\n\r\n /**\r\n * enqueue-dequeue pairs: in each iteration a thread executes an enqueue followed by a dequeue;\r\n * the benchmark executes 10^8 pairs partitioned evenly among all threads;\r\n * WARNING: If you modify this, please modify enqDeqNoTransaction() also\r\n */\r\n template\r\n uint64_t enqDeq(std::string& className, const long numPairs, const int numRuns) {\r\n nanoseconds deltas[numThreads][numRuns];\r\n atomic startFlag = { false };\r\n Q* queue = nullptr;\r\n className = Q::className();\r\n cout << \"##### \" << className << \" ##### \\n\";\r\n\r\n auto enqdeq_lambda = [this,&startFlag,&numPairs,&queue](nanoseconds *delta, const int tid) {\r\n //UserData* ud = new UserData{0,0};\r\n uint64_t* ud = new uint64_t(42);\r\n while (!startFlag.load()) {} // Spin until the startFlag is set\r\n // Warmup phase\r\n for (long long iter = 0; iter < numPairs/(numThreads*10); iter++) { // Do 1/10 iterations as warmup\r\n PTM::updateTx([=] () {\r\n queue->enqueue(*ud, tid);\r\n if (queue->dequeue(tid) == queue->EMPTY) cout << \"Error at warmup dequeueing iter=\" << iter << \"\\n\";\r\n });\r\n }\r\n // Measurement phase\r\n auto startBeats = steady_clock::now();\r\n for (long long iter = 0; iter < numPairs/numThreads; iter++) {\r\n PTM::updateTx([=] () {\r\n queue->enqueue(*ud, tid);\r\n if (queue->dequeue(tid) == queue->EMPTY) cout << \"Error at measurement dequeueing iter=\" << iter << \"\\n\";\r\n });\r\n }\r\n auto stopBeats = steady_clock::now();\r\n *delta = stopBeats - startBeats;\r\n };\r\n\r\n for (int irun = 0; irun < numRuns; irun++) {\r\n PTM::updateTx([&] () { // It's ok to capture by reference, only the main thread is active (but it is not ok for CX-PTM)\r\n queue = PTM::template tmNew();\r\n });\r\n thread enqdeqThreads[numThreads];\r\n for (int tid = 0; tid < numThreads; tid++) enqdeqThreads[tid] = thread(enqdeq_lambda, &deltas[tid][irun], tid);\r\n startFlag.store(true);\r\n // Sleep for 2 seconds just to let the threads see the startFlag\r\n this_thread::sleep_for(2s);\r\n for (int tid = 0; tid < numThreads; tid++) enqdeqThreads[tid].join();\r\n startFlag.store(false);\r\n PTM::updateTx([=] () {\r\n PTM::tmDelete(queue);\r\n });\r\n }\r\n\r\n // Sum up all the time deltas of all threads so we can find the median run\r\n vector agg(numRuns);\r\n for (int irun = 0; irun < numRuns; irun++) {\r\n agg[irun] = 0ns;\r\n for (int tid = 0; tid < numThreads; tid++) {\r\n agg[irun] += deltas[tid][irun];\r\n }\r\n }\r\n\r\n // Compute the median. numRuns should be an odd number\r\n sort(agg.begin(),agg.end());\r\n auto median = agg[numRuns/2].count()/numThreads; // Normalize back to per-thread time (mean of time for this run)\r\n\r\n cout << \"Total Ops/sec = \" << numPairs*2*NSEC_IN_SEC/median << \"\\n\";\r\n return (numPairs*2*NSEC_IN_SEC/median);\r\n }\r\n\r\n\r\n /*\r\n * WARNING: If you modify this, please modify enqDeq() also\r\n */\r\n template\r\n uint64_t enqDeqNoTransaction(std::string& className, const long numPairs, const int numRuns) {\r\n nanoseconds deltas[numThreads][numRuns];\r\n atomic startFlag = { false };\r\n Q* queue = nullptr;\r\n className = Q::className();\r\n cout << \"##### \" << className << \" ##### \\n\";\r\n\r\n auto enqdeq_lambda = [this,&startFlag,&numPairs,&queue](nanoseconds *delta, const int tid) {\r\n uint64_t* ud = new uint64_t(42);\r\n while (!startFlag.load()) {} // Spin until the startFlag is set\r\n // Warmup phase\r\n for (long long iter = 0; iter < numPairs/(numThreads*10); iter++) { // Do 1/10 iterations as warmup\r\n queue->enqueue(*ud, tid);\r\n if (queue->dequeue(tid) == queue->EMPTY) cout << \"Error at warmup dequeueing iter=\" << iter << \"\\n\";\r\n }\r\n // Measurement phase\r\n auto startBeats = steady_clock::now();\r\n for (long long iter = 0; iter < numPairs/numThreads; iter++) {\r\n queue->enqueue(*ud, tid);\r\n if (queue->dequeue(tid) == queue->EMPTY) cout << \"Error at measurement dequeueing iter=\" << iter << \"\\n\";\r\n }\r\n auto stopBeats = steady_clock::now();\r\n *delta = stopBeats - startBeats;\r\n };\r\n\r\n for (int irun = 0; irun < numRuns; irun++) {\r\n queue = new Q(); // TODO: use a PTM allocator, maybe the one in PMDK\r\n thread enqdeqThreads[numThreads];\r\n for (int tid = 0; tid < numThreads; tid++) enqdeqThreads[tid] = thread(enqdeq_lambda, &deltas[tid][irun], tid);\r\n startFlag.store(true);\r\n // Sleep for 2 seconds just to let the threads see the startFlag\r\n this_thread::sleep_for(2s);\r\n for (int tid = 0; tid < numThreads; tid++) enqdeqThreads[tid].join();\r\n startFlag.store(false);\r\n delete queue; // TODO: use PTM de-allocator\r\n }\r\n\r\n // Sum up all the time deltas of all threads so we can find the median run\r\n vector agg(numRuns);\r\n for (int irun = 0; irun < numRuns; irun++) {\r\n agg[irun] = 0ns;\r\n for (int tid = 0; tid < numThreads; tid++) {\r\n agg[irun] += deltas[tid][irun];\r\n }\r\n }\r\n\r\n // Compute the median. numRuns should be an odd number\r\n sort(agg.begin(),agg.end());\r\n auto median = agg[numRuns/2].count()/numThreads; // Normalize back to per-thread time (mean of time for this run)\r\n\r\n cout << \"Total Ops/sec = \" << numPairs*2*NSEC_IN_SEC/median << \"\\n\";\r\n return (numPairs*2*NSEC_IN_SEC/median);\r\n }\r\n\r\n\r\n\r\n /**\r\n * Start with only enqueues 100K/numThreads, wait for them to finish, then do only dequeues but only 100K/numThreads\r\n * TODO: must fix this for persistency, not yet working\r\n */\r\n template\r\n void burst(std::string& className, uint64_t& resultsEnq, uint64_t& resultsDeq,\r\n const long long burstSize, const int numIters, const int numRuns, const bool isSC=false) {\r\n Result results[numThreads][numRuns];\r\n atomic startEnq = { false };\r\n atomic startDeq = { false };\r\n atomic barrier = { 0 };\r\n Q* queue = nullptr;\r\n\r\n auto burst_lambda = [this,&startEnq,&startDeq,&burstSize,&barrier,&numIters,&isSC,&queue](Result *res, const int tid) {\r\n UserData ud(0,0);\r\n // Warmup only if it is not Single-Consumer\r\n if (!isSC) {\r\n const long long warmupIters = 100000LL; // Do 100K for each thread as a warmup\r\n for (long long iter = 0; iter < warmupIters; iter++) queue->enqueue(&ud, tid);\r\n for (long long iter = 0; iter < warmupIters; iter++) {\r\n if (queue->dequeue(tid) == nullptr) cout << \"ERROR: warmup dequeued nullptr in iter=\" << iter << \"\\n\";\r\n }\r\n }\r\n // Measurements\r\n for (int iter = 0; iter < numIters; iter++) {\r\n // Start with enqueues\r\n while (!startEnq.load()) {} // spin is better than yield here\r\n auto startBeats = steady_clock::now();\r\n for (long long i = 0; i < burstSize/numThreads; i++) {\r\n queue->enqueue(&ud, tid);\r\n }\r\n auto stopBeats = steady_clock::now();\r\n res->nsEnq += (stopBeats-startBeats);\r\n res->numEnq += burstSize/numThreads;\r\n if (barrier.fetch_add(1) == numThreads) cout << \"ERROR: in barrier\\n\";\r\n // dequeues\r\n while (!startDeq.load()) { } // spin is better than yield here\r\n if (isSC) { // Handle the single-consumer case\r\n if (tid == 0) {\r\n startBeats = steady_clock::now();\r\n // We need to deal with rounding errors in the single-consumer case\r\n for (long long i = 0; i < ((long long)(burstSize/numThreads))*numThreads; i++) {\r\n if (queue->dequeue(tid) == nullptr) {\r\n cout << \"ERROR: dequeued nullptr in iter=\" << i << \"\\n\";\r\n assert(false);\r\n }\r\n }\r\n stopBeats = steady_clock::now();\r\n if (queue->dequeue(tid) != nullptr) cout << \"ERROR: dequeued non-null, there must be duplicate items!\\n\";\r\n res->nsDeq += (stopBeats-startBeats);\r\n res->numDeq += burstSize/numThreads;\r\n }\r\n } else {\r\n startBeats = steady_clock::now();\r\n for (long long i = 0; i < burstSize/numThreads; i++) {\r\n if (queue->dequeue(tid) == nullptr) {\r\n cout << \"ERROR: dequeued nullptr in iter=\" << i << \"\\n\";\r\n assert(false);\r\n }\r\n }\r\n stopBeats = steady_clock::now();\r\n res->nsDeq += (stopBeats-startBeats);\r\n res->numDeq += burstSize/numThreads;\r\n }\r\n if (barrier.fetch_add(1) == numThreads) cout << \"ERROR: in barrier\\n\";\r\n }\r\n };\r\n\r\n for (int irun = 0; irun < numRuns; irun++) {\r\n queue = new Q(numThreads);\r\n if (irun == 0) {\r\n className = queue->className();\r\n cout << \"##### \" << queue->className() << \" ##### \\n\";\r\n }\r\n thread burstThreads[numThreads];\r\n for (int tid = 0; tid < numThreads; tid++) burstThreads[tid] = thread(burst_lambda, &results[tid][irun], tid);\r\n this_thread::sleep_for(100ms);\r\n for (int iter=0; iter < numIters; iter++) {\r\n // enqueue round\r\n startEnq.store(true);\r\n while (barrier.load() != numThreads) this_thread::yield();\r\n startEnq.store(false);\r\n long tmp = numThreads;\r\n if (!barrier.compare_exchange_strong(tmp, 0)) cout << \"ERROR: CAS\\n\";\r\n // dequeue round\r\n startDeq.store(true);\r\n while (barrier.load() != numThreads) this_thread::yield();\r\n startDeq.store(false);\r\n tmp = numThreads;\r\n if (!barrier.compare_exchange_strong(tmp, 0)) cout << \"ERROR: CAS\\n\";\r\n }\r\n for (int tid = 0; tid < numThreads; tid++) burstThreads[tid].join();\r\n delete queue;\r\n }\r\n\r\n // Accounting\r\n vector agg(numRuns);\r\n for (int irun = 0; irun < numRuns; irun++) {\r\n nanoseconds maxNsEnq = 0ns;\r\n nanoseconds maxNsDeq = 0ns;\r\n for (int tid = 0; tid < numThreads; tid++) {\r\n if (results[tid][irun].nsEnq > maxNsEnq) maxNsEnq = results[tid][irun].nsEnq;\r\n if (results[tid][irun].nsDeq > maxNsDeq) maxNsDeq = results[tid][irun].nsDeq;\r\n agg[irun].numEnq += results[tid][irun].numEnq;\r\n agg[irun].numDeq += results[tid][irun].numDeq;\r\n }\r\n agg[irun].nsEnq = maxNsEnq;\r\n agg[irun].nsDeq = maxNsDeq;\r\n agg[irun].totOpsSec = agg[irun].nsEnq.count()+agg[irun].nsDeq.count();\r\n }\r\n\r\n // Compute the median. numRuns should be an odd number\r\n sort(agg.begin(),agg.end());\r\n Result median = agg[numRuns/2];\r\n const long long allThreadsEnqPerSec = median.numEnq*NSEC_IN_SEC/median.nsEnq.count();\r\n const long long allThreadsDeqPerSec = median.numDeq*NSEC_IN_SEC/median.nsDeq.count();\r\n\r\n // Printed value is the median of the number of ops per second that all threads were able to accomplish (on average)\r\n cout << \"Enq/sec = \" << allThreadsEnqPerSec << \" Deq/sec = \" << allThreadsDeqPerSec << \"\\n\";\r\n resultsEnq = allThreadsEnqPerSec;\r\n resultsDeq = allThreadsDeqPerSec;\r\n }\r\n};\r\n\r\n#endif\r\n" }, { "path": "graphs/PBenchmarkSPS.hpp", "content": "/*\r\n * Copyright 2017-2018\r\n * Andreia Correia \r\n * Pedro Ramalhete \r\n * Pascal Felber \r\n * Nachshon Cohen \r\n *\r\n * This work is published under the MIT license. See LICENSE.txt\r\n */\r\n#ifndef _PERSISTENT_BENCHMARK_SPS_H_\r\n#define _PERSISTENT_BENCHMARK_SPS_H_\r\n\r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \r\n#include \r\n\r\nstatic const long arraySize=1000*1000; // 1M entries in the SPS array\r\n\r\nusing namespace std;\r\nusing namespace chrono;\r\n\r\n\r\n/**\r\n * This is a micro-benchmark with integer swaps (SPS) for PTMs\r\n */\r\nclass PBenchmarkSPS {\r\n\r\nprivate:\r\n int numThreads;\r\n\r\npublic:\r\n struct UserData {\r\n long long seq;\r\n int tid;\r\n UserData(long long lseq, int ltid) {\r\n this->seq = lseq;\r\n this->tid = ltid;\r\n }\r\n UserData() {\r\n this->seq = -2;\r\n this->tid = -2;\r\n }\r\n UserData(const UserData &other) : seq(other.seq), tid(other.tid) { }\r\n\r\n bool operator < (const UserData& other) const {\r\n return seq < other.seq;\r\n }\r\n };\r\n\r\n PBenchmarkSPS(int numThreads) {\r\n this->numThreads = numThreads;\r\n }\r\n\r\n\r\n /*\r\n * An array of integers that gets randomly permutated.\r\n */\r\n template class PERSIST>\r\n uint64_t benchmarkSPSInteger(std::string& className, const seconds testLengthSeconds, const long numSwapsPerTx, const int numRuns) {\r\n long long ops[numThreads][numRuns];\r\n long long lengthSec[numRuns];\r\n atomic startFlag = { false };\r\n atomic quit = { false };\r\n\r\n // Create the array of integers and initialize it, saving it in root pointer 0\r\n int larraySize = arraySize;\r\n PTM::template updateTx([larraySize] () {\r\n //PTM::pfree( PTM::template get_object>(0) ); // TODO: re-enable this after we add the clear of objects as a transaction in CX\r\n PTM::put_object(0, PTM::pmalloc( larraySize*sizeof(PERSIST*) ));\r\n return true;\r\n });\r\n // Break up the initialization into transactions of 1k stores, so it fits in the log\r\n for (long j = 0; j < arraySize; j+=1000) {\r\n PTM::template updateTx([larraySize,j] () {\r\n PERSIST* parray = PTM::template get_object>(0);\r\n for (int i = 0; i < 1000 && i+j < larraySize; i++) parray[i+j] = i+j;\r\n return true;\r\n } );\r\n }\r\n\r\n auto func = [this,&startFlag,&quit,&numSwapsPerTx](long long *ops, const int tid) {\r\n uint64_t seed = (tid*1024)+tid+1234567890123456781ULL;\r\n int larraySize = arraySize;\r\n // Spin until the startFlag is set\r\n while (!startFlag.load()) {}\r\n // Do transactions until the quit flag is set\r\n long long tcount = 0;\r\n while (!quit.load()) {\r\n // Everything has to be captured by value, or get/put in root pointers\r\n PTM::template updateTx([seed,numSwapsPerTx,larraySize] () {\r\n PERSIST* parray = PTM::template get_object>(0);\r\n uint64_t lseed = seed;\r\n for (int i = 0; i < numSwapsPerTx; i++) {\r\n lseed = randomLong(lseed);\r\n auto ia = lseed%arraySize;\r\n uint64_t tmp = parray[ia];\r\n lseed = randomLong(lseed);\r\n auto ib = lseed%arraySize;\r\n parray[ia] = parray[ib];\r\n parray[ib] = tmp;\r\n }\r\n return true;\r\n });\r\n // Can't have capture by ref for wait-free, so replicate seed advance outside tx\r\n seed = randomLong(seed);\r\n seed = randomLong(seed);\r\n ++tcount;\r\n /*\r\n PE::read_transaction([this,&seed,&parray,&numWordsPerTransaction] () {\r\n PersistentArrayInt* read_array = PE::template get_object>(PIDX_INT_ARRAY);\r\n // Check that the array is consistent\r\n int sum = 0;\r\n for (int i = 0; i < arraySize; i++) {\r\n sum += read_array->counters[i];\r\n }\r\n assert(sum == 0);\r\n } );\r\n */\r\n }\r\n *ops = tcount;\r\n };\r\n for (int irun = 0; irun < numRuns; irun++) {\r\n if (irun == 0) {\r\n className = PTM::className();\r\n cout << \"##### \" << PTM::className() << \" ##### \\n\";\r\n }\r\n thread enqdeqThreads[numThreads];\r\n for (int tid = 0; tid < numThreads; tid++) enqdeqThreads[tid] = thread(func, &ops[tid][irun], tid);\r\n auto startBeats = steady_clock::now();\r\n startFlag.store(true);\r\n // Sleep for 20 seconds\r\n this_thread::sleep_for(testLengthSeconds);\r\n quit.store(true);\r\n auto stopBeats = steady_clock::now();\r\n for (int tid = 0; tid < numThreads; tid++) enqdeqThreads[tid].join();\r\n lengthSec[irun] = (stopBeats-startBeats).count();\r\n startFlag.store(false);\r\n quit.store(false);\r\n }\r\n\r\n PTM::template updateTx([] () {\r\n PTM::pfree( PTM::template get_object>(0) );\r\n PTM::template put_object(0, nullptr);\r\n return true;\r\n });\r\n\r\n // Accounting\r\n vector agg(numRuns);\r\n for (int irun = 0; irun < numRuns; irun++) {\r\n \tfor(int i=0;i> 12; // a\r\n x ^= x << 25; // b\r\n x ^= x >> 27; // c\r\n return x * 2685821657736338717LL;\r\n }\r\n};\r\n\r\n#endif\r\n" }, { "path": "graphs/PBenchmarkSets.hpp", "content": "/*\n * Copyright 2017-2018\n * Andreia Correia \n * Pedro Ramalhete \n * Pascal Felber \n *\n * This work is published under the MIT license. See LICENSE.txt\n */\n#ifndef _PERSISTENT_BENCHMARK_SETS_H_\n#define _PERSISTENT_BENCHMARK_SETS_H_\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\nusing namespace std;\nusing namespace chrono;\n\ntemplate