[ { "path": ".github/workflows/maven-publish.yml", "content": "# This workflow will build a package using Maven and then publish it to GitHub packages when a release is created\n# For more information see: https://github.com/actions/setup-java/blob/main/docs/advanced-usage.md#apache-maven-with-a-settings-path\n\nname: Maven Package\n\non:\n release:\n types: [created]\n\njobs:\n build:\n\n runs-on: ubuntu-latest\n permissions:\n contents: read\n packages: write\n\n steps:\n - uses: actions/checkout@v2\n - name: Set up JDK 8\n uses: actions/setup-java@v2\n with:\n java-version: '8'\n distribution: 'adopt'\n server-id: github # Value of the distributionManagement/repository/id field of the pom.xml\n settings-path: ${{ github.workspace }} # location for the settings.xml file\n\n - name: Publish to GitHub Packages Apache Maven\n run: mvn deploy -s $GITHUB_WORKSPACE/settings.xml\n env:\n GITHUB_TOKEN: ${{ github.token }}\n" }, { "path": ".github/workflows/maven.yml", "content": "# This workflow will build a Java project with Maven\n# For more information see: https://help.github.com/actions/language-and-framework-guides/building-and-testing-java-with-maven\n\nname: Java CI with Maven\n\non:\n push:\n branches: [ master ]\n pull_request:\n branches: [ master ]\n\njobs:\n build:\n\n runs-on: ubuntu-latest\n\n steps:\n - uses: actions/checkout@v2\n - name: Set up JDK 8\n uses: actions/setup-java@v2\n with:\n java-version: '8'\n distribution: 'adopt'\n - name: Build with Maven\n run: mvn -B package --file pom.xml\n" }, { "path": ".gitignore", "content": "target\n.idea\nhalodb.iml\ntmp/\n" }, { "path": ".travis.yml", "content": "language: java\ndist: trusty\n\njdk:\n - oraclejdk8\n" }, { "path": "CHANGELOG.md", "content": "# HaloDB Change Log\n\n## 0.4.3 (08/20/2018)\n* Sequence number, instead of relying on system time, is now a number incremented for each write operation. \n* Include compaction rate in stats. \n\n## 0.4.2 (08/06/2018)\n* Handle the case where db crashes while it is being repaired due to error from a previous crash.\n* _put_ operation in _HaloDB_ now returns a boolean value indicating the status of the operation.\n\n## 0.4.1 (7/16/2018)\n* Include version, checksum and max file size in META file. \n* _maxFileSize_ in _HaloDBOptions_ now accepts only int values. \n\n## 0.4.0 (7/11/2018)\n* Implemented memory pool for in-memory index. \n\n" }, { "path": "CONTRIBUTING.md", "content": "# How to contribute\nFirst, thanks for taking the time to contribute to our project! The following information provides a guide for making contributions.\n\n## Code of Conduct\n\nBy participating in this project, you agree to abide by the [Oath Code of Conduct](Code-of-Conduct.md). Everyone is welcome to submit a pull request or open an issue to improve the documentation, add improvements, or report bugs.\n\n## How to Ask a Question\n\nIf you simply have a question that needs an answer, [create an issue](https://help.github.com/articles/creating-an-issue/), and label it as a question.\n\n## How To Contribute\n\n### Report a Bug or Request a Feature\n\nIf you encounter any bugs while using this software, or want to request a new feature or enhancement, feel free to [create an issue](https://help.github.com/articles/creating-an-issue/) to report it, make sure you add a label to indicate what type of issue it is.\n\n### Contribute Code\nPull requests are welcome for bug fixes. If you want to implement something new, please [request a feature first](#report-a-bug-or-request-a-feature) so we can discuss it.\n\n#### Creating a Pull Request\nPlease follow [best practices](https://github.com/trein/dev-best-practices/wiki/Git-Commit-Best-Practices) for creating git commits.\n\nWhen your code is ready to be submitted, you can [submit a pull request](https://help.github.com/articles/creating-a-pull-request/) to begin the code review process.\n" }, { "path": "CONTRIBUTORS.md", "content": " HaloDB was designed and implemented by [Arjun Mannaly](https://github.com/amannaly) " }, { "path": "Code-of-Conduct.md", "content": "# Oath Open Source Code of Conduct\n\n## Summary\nThis Code of Conduct is our way to encourage good behavior and discourage bad behavior in our open source community. We invite participation from many people to bring different perspectives to support this project. We pledge to do our part to foster a welcoming and professional environment free of harassment. We expect participants to communicate professionally and thoughtfully during their involvement with this project. \n\nParticipants may lose their good standing by engaging in misconduct. For example: insulting, threatening, or conveying unwelcome sexual content. We ask participants who observe conduct issues to report the incident directly to the project's Response Team at opensource-conduct@oath.com. Oath will assign a respondent to address the issue. We may remove harassers from this project. \n\nThis code does not replace the terms of service or acceptable use policies of the websites used to support this project. We acknowledge that participants may be subject to additional conduct terms based on their employment which may govern their online expressions.\n\n## Details\nThis Code of Conduct makes our expectations of participants in this community explicit.\n* We forbid harassment and abusive speech within this community.\n* We request participants to report misconduct to the project’s Response Team.\n* We urge participants to refrain from using discussion forums to play out a fight.\n\n### Expected Behaviors\nWe expect participants in this community to conduct themselves professionally. Since our primary mode of communication is text on an online forum (e.g. issues, pull requests, comments, emails, or chats) devoid of vocal tone, gestures, or other context that is often vital to understanding, it is important that participants are attentive to their interaction style.\n\n* **Assume positive intent.** We ask community members to assume positive intent on the part of other people’s communications. We may disagree on details, but we expect all suggestions to be supportive of the community goals.\n* **Respect participants.** We expect participants will occasionally disagree. Even if we reject an idea, we welcome everyone’s participation. Open Source projects are learning experiences. Ask, explore, challenge, and then respectfully assert if you agree or disagree. If your idea is rejected, be more persuasive not bitter.\n* **Welcoming to new members.** New members bring new perspectives. Some may raise questions that have been addressed before. Kindly point them to existing discussions. Everyone is new to every project once.\n* **Be kind to beginners.** Beginners use open source projects to get experience. They might not be talented coders yet, and projects should not accept poor quality code. But we were all beginners once, and we need to engage kindly.\n* **Consider your impact on others.** Your work will be used by others, and you depend on the work of others. We expect community members to be considerate and establish a balance their self-interest with communal interest.\n* **Use words carefully.** We may not understand intent when you say something ironic. Poe’s Law suggests that without an emoticon people will misinterpret sarcasm. We ask community members to communicate plainly.\n* **Leave with class.** When you wish to resign from participating in this project for any reason, you are free to fork the code and create a competitive project. Open Source explicitly allows this. Your exit should not be dramatic or bitter. \n\n### Unacceptable Behaviors\nParticipants remain in good standing when they do not engage in misconduct or harassment. To elaborate: \n* **Don't be a bigot.** Calling out project members by their identity or background in a negative or insulting manner. This includes, but is not limited to, slurs or insinuations related to protected or suspect classes e.g. race, color, citizenship, national origin, political belief, religion, sexual orientation, gender identity and expression, age, size, culture, ethnicity, genetic features, language, profession, national minority statue, mental or physical ability.\n* **Don't insult.** Insulting remarks about a person’s lifestyle practices.\n* **Don't dox.** Revealing private information about other participants without explicit permission.\n* **Don't intimidate.** Threats of violence or intimidation of any project member.\n* **Don't creep.** Unwanted sexual attention or content unsuited for the subject of this project.\n* **Don't disrupt.** Sustained disruptions in a discussion.\n* **Let us help.** Refusal to assist the Response Team to resolve an issue in the community.\n\nWe do not list all forms of harassment, nor imply some forms of harassment are not worthy of action. Any participant who *feels* harassed or *observes* harassment, should report the incident. Victim of harassment should not address grievances in the public forum, as this often intensifies the problem. Report it, and let us address it off-line.\n\n### Reporting Issues\nIf you experience or witness misconduct, or have any other concerns about the conduct of members of this project, please report it by contacting our Response Team at opensource-conduct@oath.com who will handle your report with discretion. Your report should include:\n* Your preferred contact information. We cannot process anonymous reports.\n* Names (real or usernames) of those involved in the incident.\n* Your account of what occurred, and if the incident is ongoing. Please provide links to or transcripts of the publicly available records (e.g. a mailing list archive or a public IRC logger), so that we can review it.\n* Any additional information that may be helpful to achieve resolution.\n\nAfter filing a report, a representative will contact you directly to review the incident and ask additional questions. If a member of the Oath Response Team is named in an incident report, that member will be recused from handling your incident. If the complaint originates from a member of the Response Team, it will be addressed by a different member of the Response Team. We will consider reports to be confidential for the purpose of protecting victims of abuse. \n\n### Scope\nOath will assign a Response Team member with admin rights on the project and legal rights on the project copyright. The Response Team is empowered to restrict some privileges to the project as needed. Since this project is governed by an open source license, any participant may fork the code under the terms of the project license. The Response Team’s goal is to preserve the project if possible, and will restrict or remove participation from those who disrupt the project. \n\nThis code does not replace the terms of service or acceptable use policies that are provided by the websites used to support this community. Nor does this code apply to communications or actions that take place outside of the context of this community. Many participants in this project are also subject to codes of conduct based on their employment. This code is a social-contract that informs participants of our social expectations. It is not a terms of service or legal contract.\n\n## License and Acknowledgment. \nThis text is shared under the [CC-BY-4.0 license](https://creativecommons.org/licenses/by/4.0/). This code is based on a study conducted by the [TODO Group](https://todogroup.org/) of many codes used in the open source community. If you have feedback about this code, contact our Response Team at the address listed above. " }, { "path": "LICENSE", "content": "Apache License\n\nVersion 2.0, January 2004\n\nhttp://www.apache.org/licenses/\n\nTERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n1. Definitions.\n\n\"License\" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document.\n\n\"Licensor\" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License.\n\n\"Legal Entity\" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, \"control\" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity.\n\n\"You\" (or \"Your\") shall mean an individual or Legal Entity exercising permissions granted by this License.\n\n\"Source\" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files.\n\n\"Object\" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types.\n\n\"Work\" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below).\n\n\"Derivative Works\" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof.\n\n\"Contribution\" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, \"submitted\" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as \"Not a Contribution.\"\n\n\"Contributor\" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work.\n\n2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form.\n\n3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed.\n\n4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions:\n\nYou must give any other recipients of the Work or Derivative Works a copy of this License; and\nYou must cause any modified files to carry prominent notices stating that You changed the files; and\nYou must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and\nIf the Work includes a \"NOTICE\" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License.\n\nYou may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License.\n5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions.\n\n6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file.\n\n7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License.\n\n8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages.\n\n9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability.\n\nEND OF TERMS AND CONDITIONS" }, { "path": "NOTICE", "content": "=========================================================================\nNOTICE file for use with, and corresponding to Section 4 of, \nthe Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0) \nin this case for the HaloDB project \n=========================================================================\n\nThis project contains software developed by Robert Stupp.\n\nOHC (https://github.com/snazy/ohc)\nJava Off-Heap-Cache, licensed under APLv2\nCopyright (C) 2014 Robert Stupp, Koeln, Germany, robert-stupp.de\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\nhttp://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License." }, { "path": "README.md", "content": "# HaloDB\n\n[![Build Status](https://travis-ci.org/yahoo/HaloDB.svg?branch=master)](https://travis-ci.org/yahoo/HaloDB)\n[![Download](https://api.bintray.com/packages/yahoo/maven/halodb/images/download.svg) ](https://bintray.com/yahoo/maven/halodb/_latestVersion)\n\nHaloDB is a fast and simple embedded key-value store written in Java. HaloDB is suitable for IO bound workloads, and is capable of handling high throughput reads and writes at submillisecond latencies. \n\nHaloDB was written for a high-throughput, low latency distributed key-value database that powers multiple ad platforms at Yahoo, therefore all its design choices and optimizations were\nprimarily for this use case. \n\nBasic design principles employed in HaloDB are not new. Refer to this [document](docs/WhyHaloDB.md) for more details about the motivation for HaloDB and its inspirations. \n \nHaloDB comprises of two main components: an index in memory which stores all the keys, and append-only log files on\nthe persistent layer which stores all the data. To reduce Java garbage collection pressure the index \nis allocated in native memory, outside the Java heap. \n\n![HaloDB](https://raw.githubusercontent.com/amannaly/HaloDB-images/master/images/halodb.png)\n\n### Basic Operations. \n```java\n // Open a db with default options.\n HaloDBOptions options = new HaloDBOptions();\n \n // Size of each data file will be 1GB.\n options.setMaxFileSize(1024 * 1024 * 1024);\n\n // Size of each tombstone file will be 64MB\n // Large file size mean less file count but will slow down db open time. But if set\n // file size too small, it will result large amount of tombstone files under db folder\n options.setMaxTombstoneFileSize(64 * 1024 * 1024);\n\n // Set the number of threads used to scan index and tombstone files in parallel\n // to build in-memory index during db open. It must be a positive number which is\n // not greater than Runtime.getRuntime().availableProcessors().\n // It is used to speed up db open time.\n options.setBuildIndexThreads(8);\n\n // The threshold at which page cache is synced to disk.\n // data will be durable only if it is flushed to disk, therefore\n // more data will be lost if this value is set too high. Setting\n // this value too low might interfere with read and write performance.\n options.setFlushDataSizeBytes(10 * 1024 * 1024);\n \n // The percentage of stale data in a data file at which the file will be compacted.\n // This value helps control write and space amplification. Increasing this value will\n // reduce write amplification but will increase space amplification.\n // This along with the compactionJobRate below is the most important setting\n // for tuning HaloDB performance. If this is set to x then write amplification \n // will be approximately 1/x. \n options.setCompactionThresholdPerFile(0.7);\n \n // Controls how fast the compaction job should run.\n // This is the amount of data which will be copied by the compaction thread per second.\n // Optimal value depends on the compactionThresholdPerFile option.\n options.setCompactionJobRate(50 * 1024 * 1024);\n \n // Setting this value is important as it helps to preallocate enough\n // memory for the off-heap cache. If the value is too low the db might\n // need to rehash the cache. For a db of size n set this value to 2*n.\n options.setNumberOfRecords(100_000_000);\n \n // Delete operation for a key will write a tombstone record to a tombstone file.\n // the tombstone record can be removed only when all previous version of that key\n // has been deleted by the compaction job.\n // enabling this option will delete during startup all tombstone records whose previous\n // versions were removed from the data file.\n options.setCleanUpTombstonesDuringOpen(true);\n \n // HaloDB does native memory allocation for the in-memory index.\n // Enabling this option will release all allocated memory back to the kernel when the db is closed.\n // This option is not necessary if the JVM is shutdown when the db is closed, as in that case\n // allocated memory is released automatically by the kernel.\n // If using in-memory index without memory pool this option,\n // depending on the number of records in the database,\n // could be a slow as we need to call _free_ for each record.\n options.setCleanUpInMemoryIndexOnClose(false);\n \n // ** settings for memory pool **\n options.setUseMemoryPool(true);\n \n // Hash table implementation in HaloDB is similar to that of ConcurrentHashMap in Java 7.\n // Hash table is divided into segments and each segment manages its own native memory.\n // The number of segments is twice the number of cores in the machine.\n // A segment's memory is further divided into chunks whose size can be configured here. \n options.setMemoryPoolChunkSize(2 * 1024 * 1024);\n \n // using a memory pool requires us to declare the size of keys in advance.\n // Any write request with key length greater than the declared value will fail, but it\n // is still possible to store keys smaller than this declared size. \n options.setFixedKeySize(8);\n \n // Represents a database instance and provides all methods for operating on the database.\n HaloDB db = null;\n \n // The directory will be created if it doesn't exist and all database files will be stored in this directory\n String directory = \"directory\";\n \n // Open the database. Directory will be created if it doesn't exist.\n // If we are opening an existing database HaloDB needs to scan all the\n // index files to create the in-memory index, which, depending on the db size, might take a few minutes.\n db = HaloDB.open(directory, options);\n \n // key and values are byte arrays. Key size is restricted to 128 bytes.\n byte[] key1 = Ints.toByteArray(200);\n byte[] value1 = \"Value for key 1\".getBytes();\n \n byte[] key2 = Ints.toByteArray(300);\n byte[] value2 = \"Value for key 2\".getBytes();\n \n // add the key-value pair to the database.\n db.put(key1, value1);\n db.put(key2, value2);\n \n // read the value from the database.\n value1 = db.get(key1);\n value2 = db.get(key2);\n \n // delete a key from the database.\n db.delete(key1);\n \n // Open an iterator and iterate through all the key-value records.\n HaloDBIterator iterator = db.newIterator();\n while (iterator.hasNext()) {\n Record record = iterator.next();\n System.out.println(Ints.fromByteArray(record.getKey()));\n System.out.println(new String(record.getValue()));\n }\n \n // get stats and print it.\n HaloDBStats stats = db.stats();\n System.out.println(stats.toString());\n \n // reset stats\n db.resetStats();\n \n // pause background compaction thread.\n // if a file is being compacted the thread\n // will block until the compaction is complete.\n db.pauseCompaction();\n \n // resume background compaction thread.\n db.resumeCompaction();\n \n // repeatedly calling pause/resume compaction methods will have no effect.\n\n // Close the database.\n db.close();\n```\nBinaries for HaloDB are hosted on [Bintray](https://bintray.com/yahoo). \n``` xml\n\n com.oath.halodb\n halodb\n x.y.x \n\n\n\n yahoo-bintray\n yahoo-bintray\n https://yahoo.bintray.com/maven\n\n``` \n \n\n\n### Read, Write and Space amplification.\nRead amplification in HaloDB is always 1—for a read request it needs to do at most one disk lookup—hence it is well suited for \nread latency critical workloads. HaloDB provides a configuration which can be tuned to control write amplification \nand space amplification, both of which trade-off with each other; HaloDB has a background compaction thread which removes stale data \nfrom the DB. The percentage of stale data at which a file is compacted can be controlled. Increasing this value will increase space amplification \nbut will reduce write amplification. For example if the value is set to 50% then write amplification will be approximately 2 \n\n\n### Durability and Crash recovery.\nWrite Ahead Logs (WAL) are usually used by databases for crash recovery. Since for HaloDB WAL _is the_ database crash recovery\nis easier and faster. \n\nHaloDB does not flush writes to disk immediately, but, for performance reasons, writes only to the OS page cache. The cache is synced to \ndisk once a configurable size is reached. In the event of a power loss, the data not flushed to disk will be lost. This compromise\nbetween performance and durability is a necessary one. \n\nIn the event of a power loss and data corruption, HaloDB will scan and discard corrupted records. Since the write thread and compaction \nthread could be writing to at most two files at a time only those files need to be repaired and hence recovery times are very short.\n\nIn the event of a power loss HaloDB offers the following consistency guarantees:\n* Writes are atomic.\n* Inserts and updates are committed to disk in the same order they are received.\n* When inserts/updates and deletes are interleaved total ordering is not guaranteed, but partial ordering is guaranteed for inserts/updates and deletes. \n \n \n### In-memory index. \nHaloDB stores all keys and their associated metadata in an index in memory. The size of this index, depending on the \nnumber and length of keys, can be quite big. Therefore, storing this in the Java Heap is a non-starter for a \nperformance critical storage engine. HaloDB solves this problem by storing the index in native memory, \noutside the heap. There are two variants of the index; one with a memory pool and the other \nwithout it. Using the memory pool helps to reduce the memory footprint of the index and reduce \nfragmentation, but requires fixed size keys. A billion 8 byte keys \ncurrently takes around 44GB of memory with memory pool and around 64GB without memory pool. \n\nThe size of the keys when using a memory pool should be declared in advance, and although this imposes an \nupper limit on the size of the keys it is still possible to store keys smaller than this declared size. \n\nWithout the memory pool, HaloDB needs to allocate native memory for every write request. Therefore, \nmemory fragmentation could be an issue. Using [jemalloc](http://jemalloc.net/) is highly recommended as it \nprovides a significant reduction in the cache's memory footprint and fragmentation.\n\n### Delete operations.\nDelete operation for a key will add a tombstone record to a tombstone file, which is distinct from the data files. \nThis design has the advantage that the tombstone record once written need not be copied again during compaction, but \nthe drawback is that in case of a power loss HaloDB cannot guarantee total ordering when put and delete operations are \ninterleaved (although partial ordering for both is guaranteed).\n\n### DB open time\nOpen db could take a few minutes, depends on number of records and tombstones. If the db open time is critical to your\nuse case, please keep tombstone file size relatively small and increase the number of threads used in building index.\nSee the option setting section in example code above. As best practice, set tombstone file size at 64MB and set build\nindex threads to number of available processors divided by number of dbs being opened simultaneously.\n\n### System requirements. \n* HaloDB requires Java 8 to run, but has not yet been tested with newer Java versions. \n* HaloDB has been tested on Linux running on x86 and on MacOS. It may run on other platforms, but this hasn't been verified yet.\n* For performance disable Transparent Huge Pages and swapping (vm.swappiness=0).\n* If a thread is interrupted JVM will close those file channels the thread was operating on.\nTherefore, don't interrupt threads while they are doing IO operations.\n\n### Restrictions. \n* Size of keys is restricted to 128 bytes. \n* HaloDB don't support range scans or ordered access.\n\n# Benchmarks.\n[Benchmarks](docs/benchmarks.md).\n \n# Contributing\nContributions are most welcome. Please refer to the [CONTRIBUTING](https://github.com/yahoo/HaloDB/blob/master/CONTRIBUTING.md) guide \n\n# Credits\nHaloDB was written by [Arjun Mannaly](https://github.com/amannaly).\n\n# License \nHaloDB is released under the Apache License, Version 2.0 \n \n " }, { "path": "benchmarks/README.md", "content": "# Storage Engine Benchmark Tool. \n\nBuild the package using **mvn clean package** This will create a far jar *target/storage-engine-benchmark-1.0.jar*\n\nDifferent benchmarks can be run using:\n \n`java -jar storage-engine-benchmark-1.0-SNAPSHOT.jar `\n\nDifferent benchmark types are defined [here](https://github.com/yahoo/HaloDB/blob/master/benchmarks/src/main/java/com/oath/halodb/benchmarks/Benchmarks.java). " }, { "path": "benchmarks/pom.xml", "content": "\n 4.0.0\n\n mannaly\n storage-engine-benchmark\n 1.0\n jar\n\n storage-engine-benchmark\n http://maven.apache.org\n\n \n UTF-8\n \n\n \n\n \n org.rocksdb\n rocksdbjni\n 5.7.2\n \n\n \n com.oath.halodb\n halodb\n 0.4.2\n \n\n \n com.fallabs\n kyotocabinet-java\n 1.16\n \n\n \n com.google.guava\n guava\n 19.0\n \n\n \n org.hdrhistogram\n HdrHistogram\n 2.1.9\n \n\n \n org.slf4j\n slf4j-simple\n 1.8.0-alpha2\n \n \n\n \n \n \n org.apache.maven.plugins\n maven-compiler-plugin\n 3.5.1\n \n 1.8\n 1.8\n \n \n\n \n \n org.apache.maven.plugins\n maven-shade-plugin\n 2.3\n \n \n \n package\n \n shade\n \n \n \n \n \n com.oath.halodb.benchmarks.BenchmarkTool\n \n \n\n \n \n *:*\n \n META-INF/*.SF\n META-INF/*.DSA\n META-INF/*.RSA\n \n \n \n\n \n \n \n \n \n \n\n \n \n halodb-bintray\n halodb-bintray\n https://yahoo.bintray.com/maven\n \n true\n \n \n false\n \n \n \n\n" }, { "path": "benchmarks/src/main/java/com/oath/halodb/benchmarks/BenchmarkTool.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\npackage com.oath.halodb.benchmarks;\n\nimport com.google.common.primitives.Longs;\nimport com.google.common.util.concurrent.RateLimiter;\n\nimport org.HdrHistogram.Histogram;\n\nimport java.io.File;\nimport java.text.DateFormat;\nimport java.text.SimpleDateFormat;\nimport java.util.Arrays;\nimport java.util.Date;\nimport java.util.Random;\nimport java.util.concurrent.TimeUnit;\n\npublic class BenchmarkTool {\n\n private static SimpleDateFormat sdf = new SimpleDateFormat(\"HH:mm:ss\");\n\n // adjust HaloDB number of records accordingly.\n private final static int numberOfRecords = 500_000_000;\n\n private static volatile boolean isReadComplete = false;\n\n private static final int numberOfReads = 640_000_000;\n private static final int numberOfReadThreads = 32;\n private static final int noOfReadsPerThread = numberOfReads / numberOfReadThreads; // 400 million.\n\n private static final int writeMBPerSecond = 20 * 1024 * 1024;\n private static final RateLimiter writeRateLimiter = RateLimiter.create(writeMBPerSecond);\n\n private static final int recordSize = 1024;\n\n private static final int seed = 100;\n private static final Random random = new Random(seed);\n\n private static RandomDataGenerator randomDataGenerator = new RandomDataGenerator(seed);\n\n public static void main(String[] args) throws Exception {\n String directoryName = args[0];\n String benchmarkType = args[1];\n Benchmarks benchmark = null;\n try {\n benchmark = Benchmarks.valueOf(benchmarkType);\n }\n catch (IllegalArgumentException e) {\n System.out.println(\"Benchmarks should be one of \" + Arrays.toString(Benchmarks.values()));\n System.exit(1);\n }\n\n System.out.println(\"Running benchmark \" + benchmark);\n\n File dir = new File(directoryName);\n\n // select different storage engines here. \n final StorageEngine db = new HaloDBStorageEngine(dir, numberOfRecords);\n //final StorageEngine db = new RocksDBStorageEngine(dir, numberOfRecords);\n //final StorageEngine db = new KyotoStorageEngine(dir, numberOfRecords);\n\n db.open();\n System.out.println(\"Opened the database.\");\n\n switch (benchmark) {\n case FILL_SEQUENCE: createDB(db, true);break;\n case FILL_RANDOM: createDB(db, false);break;\n case READ_RANDOM: readRandom(db, numberOfReadThreads);break;\n case RANDOM_UPDATE: update(db);break;\n case READ_AND_UPDATE: updateWithReads(db);\n }\n\n db.close();\n }\n\n private static void createDB(StorageEngine db, boolean isSequential) {\n long start = System.currentTimeMillis();\n byte[] value;\n long dataSize = 0;\n\n for (int i = 0; i < numberOfRecords; i++) {\n value = randomDataGenerator.getData(recordSize);\n dataSize += (long)value.length;\n\n byte[] key = isSequential ? longToBytes(i) : longToBytes(random.nextInt(numberOfRecords));\n db.put(key, value);\n\n if (i % 1_000_000 == 0) {\n System.out.printf(\"%s: Wrote %d records\\n\", DateFormat.getTimeInstance().format(new Date()), i);\n }\n }\n\n long end = System.currentTimeMillis();\n long time = (end - start) / 1000;\n System.out.println(\"Completed writing data in \" + time);\n System.out.printf(\"Write rate %d MB/sec\\n\", dataSize / time / 1024l / 1024l);\n System.out.println(\"Size of database \" + db.size());\n }\n\n private static void update(StorageEngine db) {\n long start = System.currentTimeMillis();\n byte[] value;\n long dataSize = 0;\n\n for (int i = 0; i < numberOfRecords; i++) {\n value = randomDataGenerator.getData(recordSize);\n writeRateLimiter.acquire(value.length);\n dataSize += (long)value.length;\n\n byte[] key = longToBytes(random.nextInt(numberOfRecords));\n db.put(key, value);\n\n if (i % 1_000_000 == 0) {\n System.out.printf(\"%s: Wrote %d records\\n\", DateFormat.getTimeInstance().format(new Date()), i);\n }\n }\n\n long end = System.currentTimeMillis();\n long time = (end - start) / 1000;\n System.out.println(\"Completed over writing data in \" + time);\n System.out.printf(\"Write rate %d MB/sec\\n\", dataSize / time / 1024l / 1024l);\n System.out.println(\"Size of database \" + db.size());\n }\n\n private static void readRandom(StorageEngine db, int threads) {\n Read[] reads = new Read[numberOfReadThreads];\n\n long start = System.currentTimeMillis();\n for (int i = 0; i < reads.length; i++) {\n reads[i] = new Read(db, i);\n reads[i].start();\n }\n\n for (Read r : reads) {\n try {\n r.join();\n } catch (InterruptedException e) {\n e.printStackTrace();\n }\n }\n\n long time = (System.currentTimeMillis() - start) / 1000;\n\n System.out.printf(\"Completed %d reads with %d threads in %d seconds\\n\", numberOfReads, numberOfReadThreads, time);\n System.out.println(\"Operations per second - \" + numberOfReads/time);\n\n Histogram latencyHistogram = new Histogram(TimeUnit.SECONDS.toNanos(10), 3);\n for(Read r : reads) {\n latencyHistogram.add(r.latencyHistogram);\n }\n\n System.out.printf(\"Max value - %d\\n\", latencyHistogram.getMaxValue());\n System.out.printf(\"Average value - %f\\n\", latencyHistogram.getMean());\n System.out.printf(\"95th percentile - %d\\n\", latencyHistogram.getValueAtPercentile(95.0));\n System.out.printf(\"99th percentile - %d\\n\", latencyHistogram.getValueAtPercentile(99.0));\n System.out.printf(\"99.9th percentile - %d\\n\", latencyHistogram.getValueAtPercentile(99.9));\n System.out.printf(\"99.99th percentile - %d\\n\", latencyHistogram.getValueAtPercentile(99.99));\n }\n\n private static void updateWithReads(StorageEngine db) {\n Read[] reads = new Read[numberOfReadThreads];\n\n Thread update = new Thread(new Runnable() {\n @Override\n public void run() {\n long start = System.currentTimeMillis();\n byte[] value;\n long dataSize = 0, count = 0;\n\n while (!isReadComplete) {\n value = randomDataGenerator.getData(recordSize);\n writeRateLimiter.acquire(value.length);\n dataSize += (long)value.length;\n\n byte[] key = longToBytes(random.nextInt(numberOfRecords));\n db.put(key, value);\n\n if (count++ % 1_000_000 == 0) {\n System.out.printf(\"%s: Wrote %d records\\n\", DateFormat.getTimeInstance().format(new Date()), count);\n }\n }\n\n long end = System.currentTimeMillis();\n long time = (end - start) / 1000;\n System.out.println(\"Completed over writing data in \" + time);\n System.out.println(\"Write operations per second - \" + count/time);\n System.out.printf(\"Write rate %d MB/sec\\n\", dataSize / time / 1024l / 1024l);\n System.out.println(\"Size of database \" + db.size());\n }\n });\n\n long start = System.currentTimeMillis();\n for (int i = 0; i < reads.length; i++) {\n reads[i] = new Read(db, i);\n reads[i].start();\n }\n\n update.start();\n\n for(Read r : reads) {\n try {\n r.join();\n } catch (InterruptedException e) {\n e.printStackTrace();\n }\n }\n\n long time = (System.currentTimeMillis() - start) / 1000;\n\n isReadComplete = true;\n\n long maxTime = -1;\n for (Read r : reads) {\n maxTime = Math.max(maxTime, r.time);\n }\n maxTime = maxTime / 1000;\n\n System.out.println(\"Maximum time taken by a read thread to complete - \" + maxTime);\n\n System.out.printf(\"Completed %d reads with %d threads in %d seconds\\n\", numberOfReads, numberOfReadThreads, time);\n System.out.println(\"Read operations per second - \" + numberOfReads/time);\n\n Histogram latencyHistogram = new Histogram(TimeUnit.SECONDS.toNanos(10), 3);\n for(Read r : reads) {\n latencyHistogram.add(r.latencyHistogram);\n }\n\n System.out.printf(\"Max value - %d\\n\", latencyHistogram.getMaxValue());\n System.out.printf(\"Average value - %f\\n\", latencyHistogram.getMean());\n System.out.printf(\"95th percentile - %d\\n\", latencyHistogram.getValueAtPercentile(95.0));\n System.out.printf(\"99th percentile - %d\\n\", latencyHistogram.getValueAtPercentile(99.0));\n System.out.printf(\"99.9th percentile - %d\\n\", latencyHistogram.getValueAtPercentile(99.9));\n System.out.printf(\"99.99th percentile - %d\\n\", latencyHistogram.getValueAtPercentile(99.99));\n }\n\n\n static class Read extends Thread {\n final int id;\n final Random rand;\n final StorageEngine db;\n long time;\n\n Histogram latencyHistogram = new Histogram(TimeUnit.SECONDS.toNanos(10), 3);\n\n Read(StorageEngine db, int id) {\n this.db = db;\n this.id = id;\n rand = new Random(seed + id);\n }\n\n @Override\n public void run() {\n long sum = 0, count = 0;\n long start = System.currentTimeMillis();\n\n while (count < noOfReadsPerThread) {\n long id = (long)rand.nextInt(numberOfRecords);\n long s = System.nanoTime();\n byte[] value = db.get(longToBytes(id));\n latencyHistogram.recordValue(System.nanoTime()-s);\n count++;\n if (value == null) {\n System.out.println(\"NO value for key \" +id);\n continue;\n }\n\n if (count % 1_000_000 == 0) {\n System.out.printf(printDate() + \"Read: %d Completed %d reads\\n\", this.id, count);\n }\n\n sum += value.length;\n }\n\n time = (System.currentTimeMillis() - start);\n\n System.out.printf(\"Read: %d Completed in time %d\\n\", id, time);\n }\n }\n\n public static byte[] longToBytes(long value) {\n return Longs.toByteArray(value);\n }\n\n public static String printDate() {\n return sdf.format(new Date()) + \": \";\n }\n\n\n}\n" }, { "path": "benchmarks/src/main/java/com/oath/halodb/benchmarks/Benchmarks.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb.benchmarks;\n\npublic enum Benchmarks {\n\n FILL_SEQUENCE,\n FILL_RANDOM,\n READ_RANDOM,\n RANDOM_UPDATE,\n READ_AND_UPDATE;\n\n}\n" }, { "path": "benchmarks/src/main/java/com/oath/halodb/benchmarks/HaloDBStorageEngine.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\npackage com.oath.halodb.benchmarks;\n\nimport com.google.common.primitives.Ints;\nimport com.oath.halodb.HaloDB;\nimport com.oath.halodb.HaloDBException;\nimport com.oath.halodb.HaloDBOptions;\n\nimport java.io.File;\n\npublic class HaloDBStorageEngine implements StorageEngine {\n\n private final File dbDirectory;\n\n private HaloDB db;\n private final long noOfRecords;\n\n public HaloDBStorageEngine(File dbDirectory, long noOfRecords) {\n this.dbDirectory = dbDirectory;\n this.noOfRecords = noOfRecords;\n }\n\n @Override\n public void put(byte[] key, byte[] value) {\n try {\n db.put(key, value);\n } catch (HaloDBException e) {\n e.printStackTrace();\n }\n\n }\n\n @Override\n public byte[] get(byte[] key) {\n try {\n return db.get(key);\n } catch (HaloDBException e) {\n e.printStackTrace();\n }\n\n return new byte[0];\n }\n\n @Override\n public void delete(byte[] key) {\n try {\n db.delete(key);\n } catch (HaloDBException e) {\n e.printStackTrace();\n }\n }\n\n @Override\n public void open() {\n HaloDBOptions opts = new HaloDBOptions();\n opts.setMaxFileSize(1024*1024*1024);\n opts.setCompactionThresholdPerFile(0.50);\n opts.setFlushDataSizeBytes(10 * 1024 * 1024);\n opts.setNumberOfRecords(Ints.checkedCast(2 * noOfRecords));\n opts.setCompactionJobRate(135 * 1024 * 1024);\n opts.setUseMemoryPool(true);\n opts.setFixedKeySize(8);\n\n try {\n db = HaloDB.open(dbDirectory, opts);\n } catch (HaloDBException e) {\n e.printStackTrace();\n }\n }\n\n @Override\n public void close() {\n if (db != null){\n try {\n db.close();\n } catch (HaloDBException e) {\n e.printStackTrace();\n }\n }\n\n }\n\n @Override\n public long size() {\n return db.size();\n }\n\n @Override\n public void printStats() {\n \n }\n\n @Override\n public String stats() {\n return db.stats().toString();\n }\n}\n" }, { "path": "benchmarks/src/main/java/com/oath/halodb/benchmarks/KyotoStorageEngine.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\npackage com.oath.halodb.benchmarks;\n\nimport java.io.File;\n\nimport kyotocabinet.DB;\n\npublic class KyotoStorageEngine implements StorageEngine {\n\n private final File dbDirectory;\n private final int noOfRecords;\n\n private final DB db = new DB(2);\n\n public KyotoStorageEngine(File dbDirectory, int noOfRecords) {\n this.dbDirectory = dbDirectory;\n this.noOfRecords = noOfRecords;\n }\n\n @Override\n public void open() {\n int mode = DB.OWRITER | DB.OCREATE | DB.ONOREPAIR;\n StringBuilder fileNameBuilder = new StringBuilder();\n fileNameBuilder.append(dbDirectory.getPath()).append(\"/kyoto.kch\");\n\n // specifies the power of the alignment of record size\n fileNameBuilder.append(\"#apow=\").append(8);\n // specifies the number of buckets of the hash table\n fileNameBuilder.append(\"#bnum=\").append(noOfRecords * 4);\n // specifies the mapped memory size\n fileNameBuilder.append(\"#msiz=\").append(2_500_000_000l);\n // specifies the unit step number of auto defragmentation\n fileNameBuilder.append(\"#dfunit=\").append(8);\n\n System.out.printf(\"Creating %s\\n\", fileNameBuilder.toString());\n\n if (!db.open(fileNameBuilder.toString(), mode)) {\n throw new IllegalArgumentException(String.format(\"KC db %s open error: \" + db.error(),\n fileNameBuilder.toString()));\n }\n }\n\n @Override\n public void put(byte[] key, byte[] value) {\n db.set(key, value);\n }\n\n @Override\n public byte[] get(byte[] key) {\n return db.get(key);\n }\n\n @Override\n public void close() {\n db.close();\n }\n\n @Override\n public long size() {\n return db.size();\n }\n}\n" }, { "path": "benchmarks/src/main/java/com/oath/halodb/benchmarks/RandomDataGenerator.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\npackage com.oath.halodb.benchmarks;\n\nimport java.util.Random;\n\npublic class RandomDataGenerator {\n\n private final byte[] data;\n private static final int size = 1003087;\n private int position = 0;\n\n public RandomDataGenerator(int seed) {\n this.data = new byte[size];\n Random random = new Random(seed);\n random.nextBytes(data);\n }\n\n public byte[] getData(int length) {\n byte[] b = new byte[length];\n\n for (int i = 0; i < length; i++) {\n if (position >= size) {\n position = 0;\n }\n\n b[i] = data[position++];\n }\n\n return b;\n }\n}\n" }, { "path": "benchmarks/src/main/java/com/oath/halodb/benchmarks/RocksDBStorageEngine.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\npackage com.oath.halodb.benchmarks;\n\nimport org.rocksdb.CompressionType;\nimport org.rocksdb.Env;\nimport org.rocksdb.Options;\nimport org.rocksdb.RocksDB;\nimport org.rocksdb.RocksDBException;\nimport org.rocksdb.Statistics;\nimport org.rocksdb.WriteOptions;\n\nimport java.io.File;\nimport java.util.Arrays;\nimport java.util.List;\n\npublic class RocksDBStorageEngine implements StorageEngine {\n\n private RocksDB db;\n private Options options;\n private Statistics statistics;\n private WriteOptions writeOptions;\n\n private final File dbDirectory;\n\n public RocksDBStorageEngine(File dbDirectory, int noOfRecords) {\n this.dbDirectory = dbDirectory;\n }\n\n @Override\n public void put(byte[] key, byte[] value) {\n try {\n db.put(writeOptions, key, value);\n } catch (RocksDBException e) {\n e.printStackTrace();\n }\n\n }\n\n @Override\n public byte[] get(byte[] key) {\n byte[] value = null;\n try {\n value = db.get(key);\n } catch (RocksDBException e) {\n e.printStackTrace();\n }\n\n return value;\n }\n\n @Override\n public void open() {\n options = new Options().setCreateIfMissing(true);\n options.setStatsDumpPeriodSec(1000000);\n\n options.setWriteBufferSize(128l * 1024 * 1024);\n options.setMaxWriteBufferNumber(3);\n options.setMaxBackgroundCompactions(20);\n\n Env env = Env.getDefault();\n env.setBackgroundThreads(20, Env.COMPACTION_POOL);\n options.setEnv(env);\n\n // max size of L1 10 MB.\n options.setMaxBytesForLevelBase(10485760);\n options.setTargetFileSizeBase(67108864);\n\n options.setLevel0FileNumCompactionTrigger(4);\n options.setLevel0SlowdownWritesTrigger(6);\n options.setLevel0StopWritesTrigger(12);\n options.setNumLevels(6);\n options.setDeleteObsoleteFilesPeriodMicros(300000000);\n\n\n options.setAllowMmapReads(false);\n options.setCompressionType(CompressionType.SNAPPY_COMPRESSION);\n\n\n System.out.printf(\"maxBackgroundCompactions %d \\n\", options.maxBackgroundCompactions());\n System.out.printf(\"minWriteBufferNumberToMerge %d \\n\", options.minWriteBufferNumberToMerge());\n System.out.printf(\"maxWriteBufferNumberToMaintain %d \\n\", options.maxWriteBufferNumberToMaintain());\n\n\n System.out.printf(\"level0FileNumCompactionTrigger %d \\n\", options.level0FileNumCompactionTrigger());\n System.out.printf(\"maxBytesForLevelBase %d \\n\", options.maxBytesForLevelBase());\n System.out.printf(\"maxBytesForLevelMultiplier %f \\n\", options.maxBytesForLevelMultiplier());\n System.out.printf(\"targetFileSizeBase %d \\n\", options.targetFileSizeBase());\n System.out.printf(\"targetFileSizeMultiplier %d \\n\", options.targetFileSizeMultiplier());\n\n List compressionLevels =\n Arrays.asList(\n CompressionType.NO_COMPRESSION,\n CompressionType.NO_COMPRESSION,\n CompressionType.SNAPPY_COMPRESSION,\n CompressionType.SNAPPY_COMPRESSION,\n CompressionType.SNAPPY_COMPRESSION,\n CompressionType.SNAPPY_COMPRESSION\n );\n\n options.setCompressionPerLevel(compressionLevels);\n\n System.out.printf(\"compressionPerLevel %s \\n\", options.compressionPerLevel());\n System.out.printf(\"numLevels %s \\n\", options.numLevels());\n\n writeOptions = new WriteOptions();\n writeOptions.setDisableWAL(true);\n\n System.out.printf(\"WAL is disabled - %s \\n\", writeOptions.disableWAL());\n\n try {\n db = RocksDB.open(options, dbDirectory.getPath());\n } catch (RocksDBException e) {\n e.printStackTrace();\n }\n\n }\n\n @Override\n public void close() {\n //statistics.close();\n options.close();\n writeOptions.close();\n db.close();\n }\n}\n" }, { "path": "benchmarks/src/main/java/com/oath/halodb/benchmarks/StorageEngine.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\npackage com.oath.halodb.benchmarks;\n\npublic interface StorageEngine {\n\n void put(byte[] key, byte[] value);\n\n default String stats() { return \"\";}\n\n byte[] get(byte[] key);\n\n default void delete(byte[] key) {};\n\n void open();\n\n void close();\n\n default long size() {return 0;}\n\n default void printStats() {\n\n }\n}\n" }, { "path": "docs/WhyHaloDB.md", "content": " \n# HaloDB at Yahoo.\n\nAt Yahoo, we built this high throughput, low latency distributed key-value database that runs in multiple data centers in different parts for the world. \nThe database stores billions of records and handles millions of read and write requests per second with an SLA of 1 millisecond at the 99th percentile. \n \nThe data we have in this database must be persistent, and the working set is larger than what we can fit in memory. \nTherefore, a key component of the database’s performance is a fast storage engine, for which we have relied on Kyoto Cabinet. Although Kyoto Cabinet has served us well, \nit was designed primarily for a read-heavy workload and its write throughput started to be a bottleneck as we took on more write traffic. \n \nThere were also other issues we faced with Kyoto Cabinet; it takes up to an hour to repair a corrupted db, and takes hours to iterate over and update/delete records (which we have to do every night). \nIt also doesn't expose enough operational metrics or logs which makes resolving issues challenging. However, our primary concern was Kyoto Cabinet’s write performance, \nwhich based on our projections, would have been a major obstacle for scaling the database; therefore, it was a good time to look for alternatives.\n \n**These are the salient features of the database’s workload for which the storage engine will be used:**\n* Small keys (8 bytes) and large values (10KB average)\n* Both read and write throughput are high.\n* Submillisecond read latency at the 99th percentile. \n* Single writer thread. \n* No need for ordered access or range scans.\n* Working set is much larger than available memory, hence workload is IO bound.\n* Database is written in Java.\n\n\n## Why a new storage engine?\nAlthough there are umpteen number of storage engines publicly available almost all use a variation of the following data structures to organize data on disk for fast lookup:\n* __Hash table__: Kyoto Cabinet. \n* __Log-structured merge tree__: LevelDB, RocksDB.\n* __B-Tree/B+ Tree__: Berkeley DB, InnoDB. \n\nSince our workload requires very high write throughput, Hash table and B-Tree based storage engines were not suitable as they need to do random writes. \nAlthough modern SSDs have narrowed the gap between sequential and random write performance, sequential writes still have higher throughput, primarily due \nto the reduced internal garbage collection load within the SSD. LSM trees also turned out to be unsuitable; benchmarking RocksDB on our workload showed \na write amplification of 10-12, therefore writing 100MB/sec to RocksDB meant that it will write more than 1 GB/sec to the SSD, clearly too high. \nHigh write amplification of RocksDB is a property of the LSM data structure itself, thereby ruling out storage engines based on LSM trees. \n\nLSM tree and B-Tree also maintain an ordering of keys to support efficient range scans, but the cost they pay is a read amplification greater than 1, \nand for LSM tree, very high write amplification. Since our workload only does point lookups, we don’t want to pay the cost associated with storing data \nin a format suitable for range scans. \n\nThese problems ruled out most of the publicly available and well maintained storage engines. Looking at alternate storage engine data structures led us to \nexplore ideas used in Log-structured storage systems. Here was a potential good fit; log-structured system only does sequential writes, an efficient \ngarbage collection implementation can keep write amplification low, and having an index in memory for the keys can give us a read amplification of one, \nand we get transactional updates, snapshots, and quick crash recovery almost for free. Also in this scheme, there is no ordering of data and hence its \nassociated costs are not paid. We found that similar ideas have been used in [BitCask](https://github.com/basho/bitcask/blob/develop/doc/bitcask-intro.pdf) \nand [Haystack](https://code.facebook.com/posts/685565858139515/needle-in-a-haystack-efficient-storage-of-billions-of-photos/). \nBut BitCask was written in Erlang, and since our database runs on the JVM running Erlang VM on the same box and talking to it from the JVM is something \nthat we didn’t want to do. Haystack, on the other hand, is a full-fledged distributed database optimized for storing photos, and its storage engine hasn’t been open sourced. \nTherefore it was decided to write a new storage engine from scratch; thus the HaloDB project was initiated. \n\n## Performance test results on our production workload. \nThe following chart shows the results of performance tests that we ran with production data against a performance test box with the same hardware as production boxes. The read requests were kept at 50,000 QPS while the write QPS was increased.\n\n![SSD](https://raw.githubusercontent.com/amannaly/HaloDB-images/master/images/ssd.png) \nAs you can see at the 99th percentile HaloDB read latency is an order of magnitude better than that of Kyoto Cabinet. \nWe recently upgraded our SSDs to PCIe NVMe SSDs. This has given us a significant performance boost and has narrowed the gap between HaloDB and Kyoto Cabinet, \nbut the difference is still significant:\n\n![PCIe NVMe SSD](https://raw.githubusercontent.com/amannaly/HaloDB-images/master/images/pcie-ssd.png)\n \nOf course, these are results from performance tests, but nothing beats real data from hosts running in production.\nFollowing chart shows the 99th percentile latency from a production server before and after migration to HaloDB.\n\n![99th percentile in ms](https://raw.githubusercontent.com/amannaly/HaloDB-images/master/images/before-after.png) \n \nHaloDB has thus given our production boxes a 50% improvement in capacity while consistently maintaining a sub-millisecond latency at the 99th percentile.\n \nHaloDB also has fixed few other problems that we had with KyotoCabinet. The daily cleanup job that used to take upto 5 hours in Kyoto Cabinet is now complete in 90 minutes \nwith HaloDB due to its improved write throughput. Also, HaloDB takes only a few seconds to recover from a crash due to the fact that all log files, \nonce they are rolled over, are immutable. Hence, in the event of a crash only the last file that was being written to need to be repaired. \nWhereas, with Kyoto Cabinet crash recovery used to take more than an hour to complete. And the metrics that HaloDB exposes gives us good insight into its internal state, \nwhich was missing with Kyoto Cabinet. \n \n" }, { "path": "docs/benchmarks.md", "content": "# Benchmarks \n \n Benchmarks were run to compare HaloDB against RocksDB and KyotoCabinet.\n KyotoCabinet was chosen as we were using it in production. RockDB was chosen as it is a well known storage engine\n with good documentation and a large community. HaloDB and KyotoCabinet supports only a subset of RocksDB's features, therefore the comparison is not exactly fair to RocksDB.\n \n All benchmarks were run on bare-metal box with the following specifications:\n * 2 x Xeon E5-2680 2.50GHz (HT enabled, 24 cores, 48 threads) \n * 128 GB of RAM.\n * 1 Samsung PM863 960 GB SSD with XFS file system. \n * RHEL 6 with kernel 2.6.32. \n \n \n Key size was 8 bytes and value size 1024 bytes. Tests created a db with 500 million records with total size of approximately \n 500GB. Since this is significantly bigger than the available memory it will ensure that the workload will be IO bound, which is what HaloDB was primarily designed for.\n \n Benchmark tool can be found [here](../benchmarks) \n \n## Test 1: Fill Sequential.\nCreate a new db by inserting 500 million records in sorted key order.\n![HaloDB](https://raw.githubusercontent.com/amannaly/HaloDB-images/master/images/fill-sequential.png)\n\nDB size at the end of the test run. \n\n| Storage Engine | GB |\n| ------------- | --------- |\n| HaloDB | 503 |\n| KyotoCabinet | 609 |\n| RocksDB | 487 |\n\n\n## Test 2: Random Read\nMeasure random read performance with 32 threads doing _640 million reads_ in total. Read ahead was disabled for this test. \n![HaloDB](https://raw.githubusercontent.com/amannaly/HaloDB-images/master/images/random-reads.png)\n \n## Test 3: Random Update.\nPerform 500 million updates to randomly selected records. \n![HaloDB](https://raw.githubusercontent.com/amannaly/HaloDB-images/master/images/random-update.png)\n\nDB size at the end of the test run. \n\n| Storage Engine | GB |\n| ------------- | --------- |\n| HaloDB | 556 |\n| KyotoCabinet | 609 |\n| RocksDB | 504 |\n\n## Test 4: Fill Random. \nInsert 500 million records into an empty db in random order. \n![HaloDB](https://raw.githubusercontent.com/amannaly/HaloDB-images/master/images/fill-random.png)\n\n## Test 5: Read and update. \n32 threads doing a total of 640 million random reads and one thread doing random updates as fast as possible. \n![HaloDB](https://raw.githubusercontent.com/amannaly/HaloDB-images/master/images/read-update.png)\n\n## Why HaloDB is fast.\nHaloDB doesn't claim to be always better than RocksDB or KyotoCabinet. HaloDB was written for a specific type of workload, and therefore had\nthe advantage of optimizing for that workload; the trade-offs that HaloDB makes might make it sub-optimal for other workloads (best to run benchmarks to verify). \nHaloDB also offers only a small subset of features compared to other storage engines like RocksDB. \n \nAll writes to HaloDB are sequential writes to append-only log files. HaloDB uses a background compaction job to clean up stale data. \nThe threshold at which a file is compacted can be tuned and this determines HaloDB's write amplification and space amplification. \nA compaction threshold of 50% gives a write amplification of only 2, this coupled with the fact that we do only sequential writes \nare the primary reasons for HaloDB’s high write throughput. Additionally, the only meta-data that HaloDB need to modify during writes are \nthose of the index in memory. The trade-off here is that HaloDB will occupy more space on disk. \n\nTo lookup the value for a key its corresponding metadata is first read from the in-memory index and then the value is read from disk. \nTherefore each lookup request requires at most a single read from disk, giving us a read amplification of 1, and is primarily responsible \nfor HaloDB’s low read latencies. The trade-off here is that we need to store all the keys and their associated metadata in memory. HaloDB\nalso need to scan all the keys during startup to build the in-memory index. This, depending on the number of keys, might take a few minutes. \n\nHaloDB avoids doing in-place updates and doesn't need record level locks. A type of MVCC is inherent in the design of all log-structured storage systems. This also helps with performance even under high read and write throughput.\n\nHaloDB also doesn't support range scans and therefore doesn't pay the cost associated with storing data in a format suitable for efficient range scans.\n\n" }, { "path": "pom.xml", "content": "\n\n\n 4.0.0\n\n com.oath.halodb\n halodb\n 0.5.6\n jar\n\n HaloDB\n A fast, embedded, persistent key-value storage engine.\n http://maven.apache.org\n\n \n \n Arjun Mannaly\n \n \n\n \n UTF-8\n \n\n\n \n\n \n org.slf4j\n slf4j-api\n 1.7.12\n \n\n \n com.google.guava\n guava\n 18.0\n \n\n \n net.java.dev.jna\n jna\n 4.1.0\n \n\n \n net.jpountz.lz4\n lz4\n true\n 1.3\n \n\n \n org.hamcrest\n hamcrest-all\n 1.3\n test\n \n\n \n org.apache.logging.log4j\n log4j-core\n 2.3\n test\n \n\n \n org.apache.logging.log4j\n log4j-slf4j-impl\n 2.3\n test\n \n\n \n org.testng\n testng\n 6.9.10\n test\n \n\n \n org.jmockit\n jmockit\n 1.38\n test\n \n\n \n org.assertj\n assertj-core\n 3.8.0\n test\n \n\n \n\n \n https://github.com/yahoo/HaloDB\n scm:git:git@github.com:yahoo/HaloDB.git\n HEAD\n \n\n \n \n \n org.apache.maven.plugins\n maven-compiler-plugin\n 3.5.1\n \n 1.8\n 1.8\n \n \n\n \n maven-surefire-plugin\n 2.20.1\n \n \n \n listener\n com.oath.halodb.TestListener\n \n \n -Xms2G -Xmx2G\n \n \n\n \n maven-release-plugin\n 2.5.3\n \n v@{project.version}\n \n \n\n \n\n \n \n config\n \n *.properties\n \n \n \n\n \n\n \n \n github\n https://maven.pkg.github.com/yahoo/halodb\n \n \n\n" }, { "path": "src/main/java/com/oath/halodb/CompactionManager.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.util.concurrent.RateLimiter;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport java.io.IOException;\nimport java.nio.channels.FileChannel;\nimport java.util.concurrent.BlockingQueue;\nimport java.util.concurrent.LinkedBlockingQueue;\nimport java.util.concurrent.locks.ReentrantLock;\n\nclass CompactionManager {\n private static final Logger logger = LoggerFactory.getLogger(CompactionManager.class);\n\n private final HaloDBInternal dbInternal;\n\n private volatile boolean isRunning = false;\n\n private final RateLimiter compactionRateLimiter;\n\n private volatile HaloDBFile currentWriteFile = null;\n private int currentWriteFileOffset = 0;\n\n private final BlockingQueue compactionQueue;\n\n private volatile CompactionThread compactionThread;\n\n private volatile long numberOfRecordsCopied = 0;\n private volatile long numberOfRecordsReplaced = 0;\n private volatile long numberOfRecordsScanned = 0;\n private volatile long sizeOfRecordsCopied = 0;\n private volatile long sizeOfFilesDeleted = 0;\n private volatile long totalSizeOfRecordsCopied = 0;\n private volatile long compactionStartTime = System.currentTimeMillis();\n\n private static final int STOP_SIGNAL = -10101;\n\n private final ReentrantLock startStopLock = new ReentrantLock();\n private volatile boolean stopInProgress = false;\n\n CompactionManager(HaloDBInternal dbInternal) {\n this.dbInternal = dbInternal;\n this.compactionRateLimiter = RateLimiter.create(dbInternal.options.getCompactionJobRate());\n this.compactionQueue = new LinkedBlockingQueue<>();\n }\n\n // If a file is being compacted we wait for it complete before stopping.\n boolean stopCompactionThread(boolean closeCurrentWriteFile) throws IOException {\n stopInProgress = true;\n startStopLock.lock();\n try {\n isRunning = false;\n if (isCompactionRunning()) {\n // We don't want to call interrupt on compaction thread as it\n // may interrupt IO operations and leave files in an inconsistent state.\n // instead we use -10101 as a stop signal.\n compactionQueue.put(STOP_SIGNAL);\n compactionThread.join();\n if (closeCurrentWriteFile && currentWriteFile != null) {\n currentWriteFile.flushToDisk();\n currentWriteFile.getIndexFile().flushToDisk();\n currentWriteFile.close();\n }\n }\n }\n catch (InterruptedException e) {\n logger.error(\"Error while waiting for compaction thread to stop\", e);\n return false;\n }\n finally {\n stopInProgress = false;\n startStopLock.unlock();\n }\n return true;\n }\n\n void startCompactionThread() {\n startStopLock.lock();\n try {\n if (!isCompactionRunning()) {\n isRunning = true;\n compactionThread = new CompactionThread();\n compactionThread.start();\n }\n } finally {\n startStopLock.unlock();\n }\n }\n\n void pauseCompactionThread() throws IOException {\n logger.info(\"Pausing compaction thread ...\");\n stopCompactionThread(false);\n }\n\n void resumeCompaction() {\n logger.info(\"Resuming compaction thread\");\n startCompactionThread();\n }\n\n int getCurrentWriteFileId() {\n return currentWriteFile != null ? currentWriteFile.getFileId() : -1;\n }\n\n boolean submitFileForCompaction(int fileId) {\n return compactionQueue.offer(fileId);\n }\n\n int noOfFilesPendingCompaction() {\n return compactionQueue.size();\n }\n\n long getNumberOfRecordsCopied() {\n return numberOfRecordsCopied;\n }\n\n long getNumberOfRecordsReplaced() {\n return numberOfRecordsReplaced;\n }\n\n long getNumberOfRecordsScanned() {\n return numberOfRecordsScanned;\n }\n\n long getSizeOfRecordsCopied() {\n return sizeOfRecordsCopied;\n }\n\n long getSizeOfFilesDeleted() {\n return sizeOfFilesDeleted;\n }\n\n long getCompactionJobRateSinceBeginning() {\n long timeInSeconds = (System.currentTimeMillis() - compactionStartTime)/1000;\n long rate = 0;\n if (timeInSeconds > 0) {\n rate = totalSizeOfRecordsCopied / timeInSeconds;\n }\n return rate;\n }\n\n void resetStats() {\n numberOfRecordsCopied = numberOfRecordsReplaced\n = numberOfRecordsScanned = sizeOfRecordsCopied = sizeOfFilesDeleted = 0;\n }\n\n boolean isCompactionRunning() {\n return compactionThread != null && compactionThread.isAlive();\n }\n\n private class CompactionThread extends Thread {\n\n private long unFlushedData = 0;\n\n CompactionThread() {\n super(\"CompactionThread\");\n\n setUncaughtExceptionHandler((t, e) -> {\n logger.error(\"Compaction thread crashed\", e);\n if (currentWriteFile != null) {\n try {\n currentWriteFile.flushToDisk();\n } catch (IOException ex) {\n logger.error(\"Error while flushing \" + currentWriteFile.getFileId() + \" to disk\", ex);\n }\n currentWriteFile = null;\n }\n currentWriteFileOffset = 0;\n\n if (!stopInProgress) {\n startStopLock.lock();\n try {\n compactionThread = null;\n startCompactionThread();\n } finally {\n startStopLock.unlock();\n }\n }\n else {\n logger.info(\"Not restarting thread as the lock is held by stop compaction method.\");\n }\n\n });\n }\n\n @Override\n public void run() {\n logger.info(\"Starting compaction thread ...\");\n int fileToCompact = -1;\n\n while (isRunning) {\n try {\n fileToCompact = compactionQueue.take();\n if (fileToCompact == STOP_SIGNAL) {\n logger.debug(\"Received a stop signal.\");\n // skip rest of the steps and check status of isRunning flag.\n // while pausing/stopping compaction isRunning flag must be set to false.\n continue;\n }\n logger.debug(\"Compacting {} ...\", fileToCompact);\n copyFreshRecordsToNewFile(fileToCompact);\n logger.debug(\"Completed compacting {} to {}\", fileToCompact, getCurrentWriteFileId());\n dbInternal.markFileAsCompacted(fileToCompact);\n dbInternal.deleteHaloDBFile(fileToCompact);\n }\n catch (Exception e) {\n logger.error(String.format(\"Error while compacting file %d to %d\", fileToCompact, getCurrentWriteFileId()), e);\n }\n }\n logger.info(\"Compaction thread stopped.\");\n }\n\n // TODO: group and move adjacent fresh records together for performance.\n private void copyFreshRecordsToNewFile(int idOfFileToCompact) throws IOException {\n HaloDBFile fileToCompact = dbInternal.getHaloDBFile(idOfFileToCompact);\n if (fileToCompact == null) {\n logger.debug(\"File doesn't exist, was probably compacted already.\");\n return;\n }\n\n FileChannel readFrom = fileToCompact.getChannel();\n IndexFile.IndexFileIterator iterator = fileToCompact.getIndexFile().newIterator();\n long recordsCopied = 0, recordsScanned = 0;\n\n while (iterator.hasNext()) {\n IndexFileEntry indexFileEntry = iterator.next();\n byte[] key = indexFileEntry.getKey();\n long recordOffset = indexFileEntry.getRecordOffset();\n int recordSize = indexFileEntry.getRecordSize();\n recordsScanned++;\n\n InMemoryIndexMetaData currentRecordMetaData = dbInternal.getInMemoryIndex().get(key);\n\n if (isRecordFresh(indexFileEntry, currentRecordMetaData, idOfFileToCompact)) {\n recordsCopied++;\n compactionRateLimiter.acquire(recordSize);\n rollOverCurrentWriteFile(recordSize);\n sizeOfRecordsCopied += recordSize;\n totalSizeOfRecordsCopied += recordSize;\n\n // fresh record, copy to merged file.\n long transferred = readFrom.transferTo(recordOffset, recordSize, currentWriteFile.getChannel());\n\n //TODO: for testing. remove.\n if (transferred != recordSize) {\n logger.error(\"Had to transfer {} but only did {}\", recordSize, transferred);\n }\n\n unFlushedData += transferred;\n if (dbInternal.options.getFlushDataSizeBytes() != -1 &&\n unFlushedData > dbInternal.options.getFlushDataSizeBytes()) {\n currentWriteFile.getChannel().force(false);\n unFlushedData = 0;\n }\n\n IndexFileEntry newEntry = new IndexFileEntry(\n key, recordSize, currentWriteFileOffset,\n indexFileEntry.getSequenceNumber(), indexFileEntry.getVersion(), -1\n );\n currentWriteFile.getIndexFile().write(newEntry);\n\n int valueOffset = Utils.getValueOffset(currentWriteFileOffset, key);\n InMemoryIndexMetaData newMetaData = new InMemoryIndexMetaData(\n currentWriteFile.getFileId(), valueOffset,\n currentRecordMetaData.getValueSize(), indexFileEntry.getSequenceNumber()\n );\n\n boolean updated = dbInternal.getInMemoryIndex().replace(key, currentRecordMetaData, newMetaData);\n if (updated) {\n numberOfRecordsReplaced++;\n }\n else {\n // write thread wrote a new version while this version was being compacted.\n // therefore, this version is stale.\n dbInternal.addFileToCompactionQueueIfThresholdCrossed(currentWriteFile.getFileId(), recordSize);\n }\n currentWriteFileOffset += recordSize;\n currentWriteFile.setWriteOffset(currentWriteFileOffset);\n }\n }\n\n if (recordsCopied > 0) {\n // After compaction we will delete the stale file.\n // To prevent data loss in the event of a crash we need to ensure that copied data has hit the disk.\n currentWriteFile.flushToDisk();\n }\n\n numberOfRecordsCopied += recordsCopied;\n numberOfRecordsScanned += recordsScanned;\n sizeOfFilesDeleted += fileToCompact.getSize();\n\n logger.debug(\"Scanned {} records in file {} and copied {} records to {}.datac\", recordsScanned, idOfFileToCompact, recordsCopied, getCurrentWriteFileId());\n }\n\n private boolean isRecordFresh(IndexFileEntry entry, InMemoryIndexMetaData metaData, int idOfFileToMerge) {\n return metaData != null\n && metaData.getFileId() == idOfFileToMerge\n && metaData.getValueOffset() == Utils.getValueOffset(entry.getRecordOffset(), entry.getKey());\n }\n\n private void rollOverCurrentWriteFile(int recordSize) throws IOException {\n if (currentWriteFile == null || currentWriteFileOffset + recordSize > dbInternal.options\n .getMaxFileSize()) {\n forceRolloverCurrentWriteFile();\n }\n }\n }\n\n void forceRolloverCurrentWriteFile() throws IOException {\n if (currentWriteFile != null) {\n currentWriteFile.flushToDisk();\n currentWriteFile.getIndexFile().flushToDisk();\n }\n currentWriteFile = dbInternal.createHaloDBFile(HaloDBFile.FileType.COMPACTED_FILE);\n dbInternal.getDbDirectory().syncMetaData();\n currentWriteFileOffset = 0;\n }\n\n // Used only for tests. to be called only after all writes in the test have been performed.\n @VisibleForTesting\n synchronized boolean isCompactionComplete() {\n\n if (!isCompactionRunning())\n return true;\n\n if (compactionQueue.isEmpty()) {\n try {\n isRunning = false;\n submitFileForCompaction(STOP_SIGNAL);\n compactionThread.join();\n } catch (InterruptedException e) {\n logger.error(\"Error in isCompactionComplete\", e);\n }\n\n return true;\n }\n\n return false;\n }\n}\n" }, { "path": "src/main/java/com/oath/halodb/Constants.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport java.util.regex.Pattern;\n\nclass Constants {\n\n // matches data and compacted files with extension .data and .datac respectively.\n static final Pattern DATA_FILE_PATTERN = Pattern.compile(\"([0-9]+)\" + HaloDBFile.DATA_FILE_NAME + \"c?\");\n\n static final Pattern INDEX_FILE_PATTERN = Pattern.compile(\"([0-9]+)\" + IndexFile.INDEX_FILE_NAME);\n\n static final Pattern TOMBSTONE_FILE_PATTERN = Pattern.compile(\"([0-9]+)\" + TombstoneFile.TOMBSTONE_FILE_NAME);\n\n static final Pattern STORAGE_FILE_PATTERN = Pattern.compile(\"([0-9]+)\\\\.[a-z]+\");\n}\n" }, { "path": "src/main/java/com/oath/halodb/DBDirectory.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport java.io.File;\nimport java.io.IOException;\nimport java.nio.channels.FileChannel;\nimport java.nio.file.Path;\nimport java.nio.file.StandardOpenOption;\nimport java.util.List;\n\n/**\n * Represents the top level directory for a HaloDB instance. \n */\nclass DBDirectory {\n\n private final File dbDirectory;\n private final FileChannel directoryChannel;\n\n private DBDirectory(File dbDirectory, FileChannel directoryChannel) {\n this.dbDirectory = dbDirectory;\n this.directoryChannel = directoryChannel;\n }\n\n /**\n * Will create a new directory if one doesn't already exist.\n */\n static DBDirectory open(File directory) throws IOException {\n FileUtils.createDirectoryIfNotExists(directory);\n FileChannel channel = null;\n try {\n channel = openReadOnlyChannel(directory);\n }\n catch(IOException e) {\n // only swallow the exception if its Windows\n if (!isWindows()) {\n throw e;\n }\n }\n return new DBDirectory(directory, channel);\n }\n\n void close() throws IOException {\n if (directoryChannel != null) {\n directoryChannel.close();\n }\n }\n\n Path getPath() {\n return dbDirectory.toPath();\n }\n\n File[] listDataFiles() {\n return FileUtils.listDataFiles(dbDirectory);\n }\n\n List listIndexFiles() {\n return FileUtils.listIndexFiles(dbDirectory);\n }\n\n File[] listTombstoneFiles() {\n return FileUtils.listTombstoneFiles(dbDirectory);\n }\n\n void syncMetaData() throws IOException {\n if (directoryChannel != null) {\n directoryChannel.force(true);\n }\n }\n\n /**\n * In Linux the recommended way to flush directory metadata is to open a\n * file descriptor for the directory and to call fsync on it. In Java opening a read-only file channel\n * and calling force(true) will do the same for us. But this is an undocumented behavior\n * in Java and could change in future versions.\n * https://grokbase.com/t/lucene/dev/1519kz2s50/recent-java-9-commit-e5b66323ae45-breaks-fsync-on-directory\n *\n * This currently works on Linux and OSX but may not work on other platforms. Therefore, if there is\n * an exception we silently swallow it.\n */\n private static FileChannel openReadOnlyChannel(File dbDirectory) throws IOException {\n return FileChannel.open(dbDirectory.toPath(), StandardOpenOption.READ);\n }\n\n private static boolean isWindows() {\n return System.getProperty(\"os.name\", \"generic\").toLowerCase(java.util.Locale.ENGLISH).indexOf(\"win\") != -1;\n }\n}\n" }, { "path": "src/main/java/com/oath/halodb/DBMetaData.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport java.io.IOException;\nimport java.nio.ByteBuffer;\nimport java.nio.channels.FileChannel;\nimport java.nio.channels.SeekableByteChannel;\nimport java.nio.file.*;\nimport java.util.zip.CRC32;\n\nimport static java.nio.file.StandardOpenOption.*;\nimport static java.nio.file.StandardCopyOption.*;\n\n/**\n * Represents the Metadata for the DB, stored in METADATA_FILE_NAME,\n * and contains methods to operate on it.\n */\nclass DBMetaData {\n\n /**\n * checksum - 4 bytes\n * version - 1 byte.\n * open - 1 byte\n * sequence number - 8 bytes.\n * io error - 1 byte.\n * file size - 4 byte.\n */\n private static final int META_DATA_SIZE = 4+1+1+8+1+4;\n private static final int CHECK_SUM_SIZE = 4;\n private static final int CHECK_SUM_OFFSET = 0;\n\n private long checkSum = 0;\n private int version = 0;\n private boolean open = false;\n private long sequenceNumber = 0;\n private boolean ioError = false;\n private int maxFileSize = 0;\n\n private final DBDirectory dbDirectory;\n\n static final String METADATA_FILE_NAME = \"META\";\n\n private static final Object lock = new Object();\n\n DBMetaData(DBDirectory dbDirectory) {\n this.dbDirectory = dbDirectory;\n }\n\n void loadFromFileIfExists() throws IOException {\n synchronized (lock) {\n Path metaFile = dbDirectory.getPath().resolve(METADATA_FILE_NAME);\n if (Files.exists(metaFile)) {\n try (SeekableByteChannel channel = Files.newByteChannel(metaFile)) {\n ByteBuffer buff = ByteBuffer.allocate(META_DATA_SIZE);\n channel.read(buff);\n buff.flip();\n checkSum = Utils.toUnsignedIntFromInt(buff.getInt());\n version = Utils.toUnsignedByte(buff.get());\n open = buff.get() != 0;\n sequenceNumber = buff.getLong();\n ioError = buff.get() != 0;\n maxFileSize = buff.getInt();\n }\n }\n }\n }\n\n void storeToFile() throws IOException {\n synchronized (lock) {\n String tempFileName = METADATA_FILE_NAME + \".temp\";\n Path tempFile = dbDirectory.getPath().resolve(tempFileName);\n Files.deleteIfExists(tempFile);\n try(FileChannel channel = FileChannel.open(tempFile, WRITE, CREATE, SYNC)) {\n ByteBuffer buff = ByteBuffer.allocate(META_DATA_SIZE);\n buff.position(CHECK_SUM_SIZE);\n buff.put((byte)version);\n buff.put((byte)(open ? 0xFF : 0));\n buff.putLong(sequenceNumber);\n buff.put((byte)(ioError ? 0xFF : 0));\n buff.putInt(maxFileSize);\n\n long crc32 = computeCheckSum(buff.array());\n buff.putInt(CHECK_SUM_OFFSET, (int)crc32);\n\n buff.flip();\n channel.write(buff);\n Files.move(tempFile, dbDirectory.getPath().resolve(METADATA_FILE_NAME), REPLACE_EXISTING, ATOMIC_MOVE);\n dbDirectory.syncMetaData();\n }\n }\n }\n\n private long computeCheckSum(byte[] header) {\n CRC32 crc32 = new CRC32();\n crc32.update(header, CHECK_SUM_OFFSET + CHECK_SUM_SIZE, META_DATA_SIZE - CHECK_SUM_SIZE);\n return crc32.getValue();\n }\n\n boolean isValid() {\n ByteBuffer buff = ByteBuffer.allocate(META_DATA_SIZE);\n buff.position(CHECK_SUM_SIZE);\n buff.put((byte)version);\n buff.put((byte)(open ? 0xFF : 0));\n buff.putLong(sequenceNumber);\n buff.put((byte)(ioError ? 0xFF : 0));\n buff.putInt(maxFileSize);\n\n return computeCheckSum(buff.array()) == checkSum;\n }\n\n boolean isOpen() {\n return open;\n }\n\n void setOpen(boolean open) {\n this.open = open;\n }\n\n long getSequenceNumber() {\n return sequenceNumber;\n }\n\n void setSequenceNumber(long sequenceNumber) {\n this.sequenceNumber = sequenceNumber;\n }\n\n boolean isIOError() {\n return ioError;\n }\n\n void setIOError(boolean ioError) {\n this.ioError = ioError;\n }\n\n public int getVersion() {\n return version;\n }\n\n public void setVersion(int version) {\n this.version = version;\n }\n\n public int getMaxFileSize() {\n return maxFileSize;\n }\n\n public void setMaxFileSize(int maxFileSize) {\n this.maxFileSize = maxFileSize;\n }\n}\n" }, { "path": "src/main/java/com/oath/halodb/FileUtils.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport java.io.File;\nimport java.io.IOException;\nimport java.nio.file.FileSystem;\nimport java.nio.file.Files;\nimport java.nio.file.Path;\nimport java.nio.file.spi.FileSystemProvider;\nimport java.util.Arrays;\nimport java.util.Collections;\nimport java.util.Comparator;\nimport java.util.List;\nimport java.util.regex.Matcher;\nimport java.util.regex.Pattern;\nimport java.util.stream.Collectors;\n\nclass FileUtils {\n\n static void createDirectoryIfNotExists(File directory) throws IOException {\n if (directory.exists()) {\n if (!directory.isDirectory()) {\n throw new IOException(directory.getName() + \" is not a directory.\");\n }\n return;\n }\n\n if (!directory.mkdirs()) {\n throw new IOException(\"Cannot create directory \" + directory.getName());\n }\n }\n\n static void deleteDirectory(File dir) throws IOException {\n File[] files = dir.listFiles();\n if (files != null) {\n for (File file : files) {\n if (file.isDirectory()) {\n deleteDirectory(file);\n } else {\n Files.delete(file.toPath());\n }\n }\n }\n\n Files.deleteIfExists(dir.toPath());\n }\n\n static List listIndexFiles(File directory) {\n File[] files = directory.listFiles(file -> Constants.INDEX_FILE_PATTERN.matcher(file.getName()).matches());\n if (files == null)\n return Collections.emptyList();\n\n // sort in ascending order. we want the earliest index files to be processed first.\n return Arrays.stream(files)\n .sorted(Comparator.comparingInt(f -> getFileId(f, Constants.INDEX_FILE_PATTERN)))\n .map(f -> getFileId(f, Constants.INDEX_FILE_PATTERN))\n .collect(Collectors.toList());\n }\n\n /**\n * Returns all *.tombstone files in the given directory sorted by file id.\n */\n static File[] listTombstoneFiles(File directory) {\n File[] files = directory.listFiles(file -> Constants.TOMBSTONE_FILE_PATTERN.matcher(file.getName()).matches());\n if (files == null)\n return new File[0];\n\n Comparator comparator = Comparator.comparingInt(f -> getFileId(f, Constants.TOMBSTONE_FILE_PATTERN));\n Arrays.sort(files, comparator);\n return files;\n }\n\n /**\n * Returns all *.data and *.datac files in the given directory.\n */\n static File[] listDataFiles(File directory) {\n return directory.listFiles(file -> Constants.DATA_FILE_PATTERN.matcher(file.getName()).matches());\n }\n\n private static int getFileId(File file, Pattern pattern) {\n Matcher matcher = pattern.matcher(file.getName());\n if (matcher.find()) {\n return Integer.valueOf(matcher.group(1));\n }\n\n throw new IllegalArgumentException(\"Cannot extract file id for file \" + file.getPath());\n\n }\n}\n" }, { "path": "src/main/java/com/oath/halodb/HaloDB.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport com.google.common.annotations.VisibleForTesting;\n\nimport java.io.File;\nimport java.io.IOException;\nimport java.util.Set;\n\npublic final class HaloDB {\n\n private HaloDBInternal dbInternal;\n\n private File directory;\n\n public static HaloDB open(File dirname, HaloDBOptions opts) throws HaloDBException {\n HaloDB db = new HaloDB();\n try {\n db.dbInternal = HaloDBInternal.open(dirname, opts);\n db.directory = dirname;\n } catch (IOException e) {\n throw new HaloDBException(\"Failed to open db \" + dirname.getName(), e);\n }\n return db;\n }\n\n public static HaloDB open(String directory, HaloDBOptions opts) throws HaloDBException {\n return HaloDB.open(new File(directory), opts);\n }\n\n public byte[] get(byte[] key) throws HaloDBException {\n try {\n return dbInternal.get(key, 1);\n } catch (IOException e) {\n throw new HaloDBException(\"Lookup failed.\", e);\n }\n }\n\n public boolean put(byte[] key, byte[] value) throws HaloDBException {\n try {\n return dbInternal.put(key, value);\n } catch (IOException e) {\n throw new HaloDBException(\"Store to db failed.\", e);\n }\n }\n\n public void delete(byte[] key) throws HaloDBException {\n try {\n dbInternal.delete(key);\n } catch (IOException e) {\n throw new HaloDBException(\"Delete operation failed.\", e);\n }\n }\n\n public void close() throws HaloDBException {\n try {\n dbInternal.close();\n } catch (IOException e) {\n throw new HaloDBException(\"Error while closing \" + directory.getName(), e);\n }\n }\n\n public long size() {\n return dbInternal.size();\n }\n\n public HaloDBStats stats() {\n return dbInternal.stats();\n }\n\n public void resetStats() {\n dbInternal.resetStats();\n }\n\n public HaloDBIterator newIterator() throws HaloDBException {\n return new HaloDBIterator(dbInternal);\n }\n\n public HaloDBKeyIterator newKeyIterator() {\n return new HaloDBKeyIterator(dbInternal);\n }\n\n public void pauseCompaction() throws HaloDBException {\n try {\n dbInternal.pauseCompaction();\n } catch (IOException e) {\n throw new HaloDBException(\"Error while trying to pause compaction thread\", e);\n }\n }\n\n public boolean snapshot() {\n return dbInternal.takeSnapshot();\n }\n\n public boolean clearSnapshot() {\n return dbInternal.clearSnapshot();\n }\n\n public File getSnapshotDirectory() {\n return dbInternal.getSnapshotDirectory();\n }\n\n public void resumeCompaction() {\n dbInternal.resumeCompaction();\n }\n\n // methods used in tests.\n\n @VisibleForTesting\n boolean isCompactionComplete() {\n return dbInternal.isCompactionComplete();\n }\n\n @VisibleForTesting\n boolean isTombstoneFilesMerging() {\n return dbInternal.isTombstoneFilesMerging();\n }\n}\n" }, { "path": "src/main/java/com/oath/halodb/HaloDBException.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\npackage com.oath.halodb;\n\npublic class HaloDBException extends Exception {\n private static final long serialVersionUID = 1010101L;\n\n public HaloDBException(String message) {\n super(message);\n }\n\n public HaloDBException(String message, Throwable cause) {\n super(message, cause);\n }\n\n public HaloDBException(Throwable cause) {\n super(cause);\n }\n}\n" }, { "path": "src/main/java/com/oath/halodb/HaloDBFile.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport com.google.common.primitives.Ints;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport java.io.File;\nimport java.io.IOException;\nimport java.io.RandomAccessFile;\nimport java.nio.ByteBuffer;\nimport java.nio.channels.FileChannel;\nimport java.nio.file.Files;\nimport java.nio.file.Path;\nimport java.util.Iterator;\nimport java.util.function.BiFunction;\nimport java.util.regex.Matcher;\n\nimport static java.nio.file.StandardCopyOption.ATOMIC_MOVE;\nimport static java.nio.file.StandardCopyOption.REPLACE_EXISTING;\n\n/**\n * Represents a data file and its associated index file.\n */\nclass HaloDBFile {\n private static final Logger logger = LoggerFactory.getLogger(HaloDBFile.class);\n\n private volatile int writeOffset;\n\n private FileChannel channel;\n private final File backingFile;\n private final DBDirectory dbDirectory;\n private final int fileId;\n\n private IndexFile indexFile;\n\n private final HaloDBOptions options;\n\n private long unFlushedData = 0;\n\n static final String DATA_FILE_NAME = \".data\";\n static final String COMPACTED_DATA_FILE_NAME = \".datac\";\n\n private final FileType fileType;\n\n private HaloDBFile(int fileId, File backingFile, DBDirectory dbDirectory, IndexFile indexFile, FileType fileType,\n FileChannel channel, HaloDBOptions options) throws IOException {\n this.fileId = fileId;\n this.backingFile = backingFile;\n this.dbDirectory = dbDirectory;\n this.indexFile = indexFile;\n this.fileType = fileType;\n this.channel = channel;\n this.writeOffset = Ints.checkedCast(channel.size());\n this.options = options;\n }\n\n byte[] readFromFile(int offset, int length) throws IOException {\n byte[] value = new byte[length];\n ByteBuffer valueBuf = ByteBuffer.wrap(value);\n int read = readFromFile(offset, valueBuf);\n assert read == length;\n\n return value;\n }\n\n int readFromFile(long position, ByteBuffer destinationBuffer) throws IOException {\n long currentPosition = position;\n int bytesRead;\n do {\n bytesRead = channel.read(destinationBuffer, currentPosition);\n currentPosition += bytesRead;\n } while (bytesRead != -1 && destinationBuffer.hasRemaining());\n\n return (int)(currentPosition - position);\n }\n\n private Record readRecord(int offset) throws HaloDBException, IOException {\n long tempOffset = offset;\n\n // read the header from disk.\n ByteBuffer headerBuf = ByteBuffer.allocate(Record.Header.HEADER_SIZE);\n int readSize = readFromFile(offset, headerBuf);\n if (readSize != Record.Header.HEADER_SIZE) {\n throw new HaloDBException(\"Corrupted header at \" + offset + \" in file \" + fileId);\n }\n tempOffset += readSize;\n\n Record.Header header = Record.Header.deserialize(headerBuf);\n if (!Record.Header.verifyHeader(header)) {\n throw new HaloDBException(\"Corrupted header at \" + offset + \" in file \" + fileId);\n }\n\n // read key-value from disk.\n ByteBuffer recordBuf = ByteBuffer.allocate(header.getKeySize() + header.getValueSize());\n readSize = readFromFile(tempOffset, recordBuf);\n if (readSize != recordBuf.capacity()) {\n throw new HaloDBException(\"Corrupted record at \" + offset + \" in file \" + fileId);\n }\n\n Record record = Record.deserialize(recordBuf, header.getKeySize(), header.getValueSize());\n record.setHeader(header);\n int valueOffset = offset + Record.Header.HEADER_SIZE + header.getKeySize();\n record.setRecordMetaData(new InMemoryIndexMetaData(fileId, valueOffset, header.getValueSize(), header.getSequenceNumber()));\n return record;\n }\n\n InMemoryIndexMetaData writeRecord(Record record) throws IOException {\n writeToChannel(record.serialize());\n\n int recordSize = record.getRecordSize();\n int recordOffset = writeOffset;\n writeOffset += recordSize;\n\n IndexFileEntry indexFileEntry = new IndexFileEntry(\n record.getKey(), recordSize,\n recordOffset, record.getSequenceNumber(),\n Versions.CURRENT_INDEX_FILE_VERSION, -1\n );\n indexFile.write(indexFileEntry);\n\n int valueOffset = Utils.getValueOffset(recordOffset, record.getKey());\n return new InMemoryIndexMetaData(fileId, valueOffset, record.getValue().length, record.getSequenceNumber());\n }\n\n void rebuildIndexFile() throws IOException {\n indexFile.delete();\n\n indexFile = new IndexFile(fileId, dbDirectory, options);\n indexFile.create();\n\n HaloDBFileIterator iterator = new HaloDBFileIterator();\n int offset = 0;\n while (iterator.hasNext()) {\n Record record = iterator.next();\n IndexFileEntry indexFileEntry = new IndexFileEntry(\n record.getKey(), record.getRecordSize(),\n offset, record.getSequenceNumber(),\n Versions.CURRENT_INDEX_FILE_VERSION, -1\n );\n indexFile.write(indexFileEntry);\n offset += record.getRecordSize();\n }\n }\n\n /**\n * Copies to a temporary file those records whose computed checksum matches the stored one and then atomically\n * rename the temp file to the current file.\n * Records in the file which occur after a corrupted record are discarded.\n * Index file is also recreated.\n * This method is called if we detect an unclean shutdown.\n */\n HaloDBFile repairFile(DBDirectory dbDirectory) throws IOException {\n HaloDBFile repairFile = createRepairFile();\n\n logger.info(\"Repairing file {}.\", getName());\n HaloDBFileIterator iterator = new HaloDBFileIterator();\n int count = 0;\n while (iterator.hasNext()) {\n Record record = iterator.next();\n // if the header is corrupted iterator will return null.\n if (record != null && record.verifyChecksum()) {\n repairFile.writeRecord(record);\n count++;\n }\n else {\n logger.info(\"Found a corrupted record after copying {} records\", count);\n break;\n }\n }\n logger.info(\"Recovered {} records from file {} with size {}. Size after repair {}.\", count, getName(), getSize(), repairFile.getSize());\n repairFile.flushToDisk();\n repairFile.indexFile.flushToDisk();\n Files.move(repairFile.indexFile.getPath(), indexFile.getPath(), REPLACE_EXISTING, ATOMIC_MOVE);\n Files.move(repairFile.getPath(), getPath(), REPLACE_EXISTING, ATOMIC_MOVE);\n dbDirectory.syncMetaData();\n repairFile.close();\n close();\n return openForReading(dbDirectory, getPath().toFile(), fileType, options);\n }\n\n private HaloDBFile createRepairFile() throws IOException {\n File repairFile = dbDirectory.getPath().resolve(getName()+\".repair\").toFile();\n while (!repairFile.createNewFile()) {\n logger.info(\"Repair file {} already exists, probably from a previous repair which failed. Deleting and trying again\", repairFile.getName());\n repairFile.delete();\n }\n\n FileChannel channel = new RandomAccessFile(repairFile, \"rw\").getChannel();\n IndexFile indexFile = new IndexFile(fileId, dbDirectory, options);\n indexFile.createRepairFile();\n return new HaloDBFile(fileId, repairFile, dbDirectory, indexFile, fileType, channel, options);\n }\n\n private long writeToChannel(ByteBuffer[] buffers) throws IOException {\n long toWrite = 0;\n for (ByteBuffer buffer : buffers) {\n toWrite += buffer.remaining();\n }\n\n long written = 0;\n while (written < toWrite) {\n written += channel.write(buffers);\n }\n\n unFlushedData += written;\n\n if (options.isSyncWrite() || (options.getFlushDataSizeBytes() != -1 && unFlushedData > options.getFlushDataSizeBytes())) {\n flushToDisk();\n unFlushedData = 0;\n }\n return written;\n }\n\n void flushToDisk() throws IOException {\n if (channel != null && channel.isOpen())\n channel.force(true);\n }\n\n long getWriteOffset() {\n return writeOffset;\n }\n\n void setWriteOffset(int writeOffset) {\n this.writeOffset = writeOffset;\n }\n\n long getSize() {\n return writeOffset;\n }\n\n IndexFile getIndexFile() {\n return indexFile;\n }\n\n FileChannel getChannel() {\n return channel;\n }\n\n FileType getFileType() {\n return fileType;\n }\n\n int getFileId() {\n return fileId;\n }\n\n static HaloDBFile openForReading(DBDirectory dbDirectory, File filename, FileType fileType, HaloDBOptions options) throws IOException {\n int fileId = HaloDBFile.getFileTimeStamp(filename);\n FileChannel channel = new RandomAccessFile(filename, \"r\").getChannel();\n IndexFile indexFile = new IndexFile(fileId, dbDirectory, options);\n indexFile.open();\n\n return new HaloDBFile(fileId, filename, dbDirectory, indexFile, fileType, channel, options);\n }\n\n static HaloDBFile create(DBDirectory dbDirectory, int fileId, HaloDBOptions options, FileType fileType) throws IOException {\n BiFunction toFile = (fileType == FileType.DATA_FILE) ? HaloDBFile::getDataFile : HaloDBFile::getCompactedDataFile;\n\n File file = toFile.apply(dbDirectory, fileId);\n while (!file.createNewFile()) {\n // file already exists try another one.\n fileId++;\n file = toFile.apply(dbDirectory, fileId);\n }\n\n FileChannel channel = new RandomAccessFile(file, \"rw\").getChannel();\n //TODO: setting the length might improve performance.\n //file.setLength(max_);\n\n IndexFile indexFile = new IndexFile(fileId, dbDirectory, options);\n indexFile.create();\n\n return new HaloDBFile(fileId, file, dbDirectory, indexFile, fileType, channel, options);\n }\n\n HaloDBFileIterator newIterator() throws IOException {\n return new HaloDBFileIterator();\n }\n\n void close() throws IOException {\n if (channel != null) {\n channel.close();\n }\n if (indexFile != null) {\n indexFile.close();\n }\n }\n\n void delete() throws IOException {\n close();\n if (backingFile != null)\n backingFile.delete();\n\n if (indexFile != null)\n indexFile.delete();\n }\n\n String getName() {\n return backingFile.getName();\n }\n\n Path getPath() {\n return backingFile.toPath();\n }\n\n private static File getDataFile(DBDirectory dbDirectory, int fileId) {\n return dbDirectory.getPath().resolve(fileId + DATA_FILE_NAME).toFile();\n }\n\n private static File getCompactedDataFile(DBDirectory dbDirectory, int fileId) {\n return dbDirectory.getPath().resolve(fileId + COMPACTED_DATA_FILE_NAME).toFile();\n }\n\n static FileType findFileType(File file) {\n String name = file.getName();\n return name.endsWith(COMPACTED_DATA_FILE_NAME) ? FileType.COMPACTED_FILE : FileType.DATA_FILE;\n }\n\n static int getFileTimeStamp(File file) {\n Matcher matcher = Constants.DATA_FILE_PATTERN.matcher(file.getName());\n matcher.find();\n String s = matcher.group(1);\n return Integer.parseInt(s);\n }\n\n /**\n * This iterator is intended only to be used internally as it behaves bit differently\n * from expected Iterator behavior: If a record is corrupted next() will return null although hasNext()\n * returns true.\n */\n class HaloDBFileIterator implements Iterator {\n\n private final int endOffset;\n private int currentOffset = 0;\n\n HaloDBFileIterator() throws IOException {\n this.endOffset = Ints.checkedCast(channel.size());\n }\n\n @Override\n public boolean hasNext() {\n return currentOffset < endOffset;\n }\n\n @Override\n public Record next() {\n Record record;\n try {\n record = readRecord(currentOffset);\n } catch (IOException | HaloDBException e) {\n // we have encountered an error, probably because record is corrupted.\n // we skip rest of the file and return null.\n logger.error(\"Error in iterator\", e);\n currentOffset = endOffset;\n return null;\n }\n currentOffset += record.getRecordSize();\n return record;\n }\n }\n\n enum FileType {\n DATA_FILE, COMPACTED_FILE;\n }\n}\n" }, { "path": "src/main/java/com/oath/halodb/HaloDBInternal.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.primitives.Ints;\n\nimport com.google.common.util.concurrent.RateLimiter;\nimport java.nio.file.Files;\nimport java.nio.file.Path;\nimport java.nio.file.Paths;\nimport java.util.regex.*;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport java.io.File;\nimport java.io.IOException;\nimport java.nio.ByteBuffer;\nimport java.nio.channels.ClosedChannelException;\nimport java.nio.channels.FileChannel;\nimport java.nio.channels.FileLock;\nimport java.nio.channels.OverlappingFileLockException;\nimport java.nio.file.StandardOpenOption;\nimport java.util.ArrayList;\nimport java.util.Comparator;\nimport java.util.HashMap;\nimport java.util.HashSet;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.Optional;\nimport java.util.Set;\nimport java.util.concurrent.*;\nimport java.util.concurrent.atomic.AtomicInteger;\nimport java.util.concurrent.atomic.AtomicLong;\nimport java.util.concurrent.locks.Lock;\nimport java.util.concurrent.locks.ReentrantLock;\n\nclass HaloDBInternal {\n\n private static final Logger logger = LoggerFactory.getLogger(HaloDBInternal.class);\n static final String SNAPSHOT_SUBDIR = \"snapshot\";\n\n private DBDirectory dbDirectory;\n\n private volatile HaloDBFile currentWriteFile;\n\n private volatile TombstoneFile currentTombstoneFile;\n\n private volatile Thread tombstoneMergeThread;\n\n private Map readFileMap = new ConcurrentHashMap<>();\n\n HaloDBOptions options;\n\n private InMemoryIndex inMemoryIndex;\n\n private final Map staleDataPerFileMap = new ConcurrentHashMap<>();\n\n private CompactionManager compactionManager;\n\n private AtomicInteger nextFileId;\n\n private volatile boolean isClosing = false;\n\n private volatile long statsResetTime = System.currentTimeMillis();\n\n private FileLock dbLock;\n\n private final Lock writeLock = new ReentrantLock();\n\n private static final int maxReadAttempts = 5;\n\n private AtomicLong noOfTombstonesCopiedDuringOpen;\n private AtomicLong noOfTombstonesFoundDuringOpen;\n private volatile long nextSequenceNumber;\n\n private volatile boolean isTombstoneFilesMerging = false;\n\n private HaloDBInternal() {}\n\n static HaloDBInternal open(File directory, HaloDBOptions options) throws HaloDBException, IOException {\n checkIfOptionsAreCorrect(options);\n\n HaloDBInternal dbInternal = new HaloDBInternal();\n try {\n dbInternal.dbDirectory = DBDirectory.open(directory);\n dbInternal.dbLock = dbInternal.getLock();\n dbInternal.options = options;\n\n int maxFileId = dbInternal.buildReadFileMap();\n dbInternal.nextFileId = new AtomicInteger(maxFileId + 10);\n\n dbInternal.noOfTombstonesCopiedDuringOpen = new AtomicLong(0);\n dbInternal.noOfTombstonesFoundDuringOpen = new AtomicLong(0);\n\n DBMetaData dbMetaData = new DBMetaData(dbInternal.dbDirectory);\n dbMetaData.loadFromFileIfExists();\n if (dbMetaData.getMaxFileSize() != 0 && dbMetaData.getMaxFileSize() != options.getMaxFileSize()) {\n throw new IllegalArgumentException(\"File size cannot be changed after db was created. Current size \" + dbMetaData.getMaxFileSize());\n }\n\n if (dbMetaData.isOpen() || dbMetaData.isIOError()) {\n logger.info(\"DB was not shutdown correctly last time. Files may not be consistent, repairing them.\");\n // open flag is true, this might mean that the db was not cleanly closed the last time.\n dbInternal.repairFiles();\n }\n dbMetaData.setOpen(true);\n dbMetaData.setIOError(false);\n dbMetaData.setVersion(Versions.CURRENT_META_FILE_VERSION);\n dbMetaData.setMaxFileSize(options.getMaxFileSize());\n dbMetaData.storeToFile();\n\n dbInternal.compactionManager = new CompactionManager(dbInternal);\n\n dbInternal.inMemoryIndex = new InMemoryIndex(\n options.getNumberOfRecords(), options.isUseMemoryPool(),\n options.getFixedKeySize(), options.getMemoryPoolChunkSize()\n );\n\n long maxSequenceNumber = dbInternal.buildInMemoryIndex();\n if (maxSequenceNumber == -1L) {\n dbInternal.nextSequenceNumber = 1;\n logger.info(\"Didn't find any existing records; initializing max sequence number to 1\");\n } else {\n dbInternal.nextSequenceNumber = maxSequenceNumber + 100;\n logger.info(\"Found max sequence number {}, now starting from {}\", maxSequenceNumber, dbInternal.nextSequenceNumber);\n }\n\n if (!options.isCompactionDisabled()) {\n dbInternal.compactionManager.startCompactionThread();\n }\n else {\n logger.warn(\"Compaction is disabled in HaloDBOption. This should happen only in tests\");\n }\n\n // merge tombstone files at background if clean up set to true\n if (options.isCleanUpTombstonesDuringOpen()) {\n dbInternal.isTombstoneFilesMerging = true;\n dbInternal.tombstoneMergeThread = new Thread(() -> { dbInternal.mergeTombstoneFiles(); });\n dbInternal.tombstoneMergeThread.start();\n }\n\n logger.info(\"Opened HaloDB {}\", directory.getName());\n logger.info(\"maxFileSize - {}\", options.getMaxFileSize());\n logger.info(\"compactionThresholdPerFile - {}\", options.getCompactionThresholdPerFile());\n } catch (Exception e) {\n // release the lock if open() failed.\n if (dbInternal.dbLock != null) {\n dbInternal.dbLock.close();\n }\n throw e;\n }\n\n return dbInternal;\n }\n\n synchronized void close() throws IOException {\n writeLock.lock();\n try {\n if (isClosing) {\n // instance already closed.\n return;\n }\n isClosing = true;\n\n try {\n if(!compactionManager.stopCompactionThread(true))\n setIOErrorFlag();\n } catch (IOException e) {\n logger.error(\"Error while stopping compaction thread. Setting IOError flag\", e);\n setIOErrorFlag();\n }\n\n if (isTombstoneFilesMerging) {\n try {\n tombstoneMergeThread.join();\n } catch (InterruptedException e) {\n logger.error(\"Interrupted when waiting the tombstone files merging\");\n setIOErrorFlag();\n }\n }\n\n if (options.isCleanUpInMemoryIndexOnClose())\n inMemoryIndex.close();\n\n if (currentWriteFile != null) {\n currentWriteFile.flushToDisk();\n currentWriteFile.getIndexFile().flushToDisk();\n currentWriteFile.close();\n }\n if (currentTombstoneFile != null) {\n currentTombstoneFile.flushToDisk();\n currentTombstoneFile.close();\n }\n\n for (HaloDBFile file : readFileMap.values()) {\n file.close();\n }\n\n DBMetaData metaData = new DBMetaData(dbDirectory);\n metaData.loadFromFileIfExists();\n metaData.setOpen(false);\n metaData.storeToFile();\n\n dbDirectory.close();\n\n if (dbLock != null) {\n dbLock.close();\n }\n } finally {\n writeLock.unlock();\n }\n }\n\n boolean put(byte[] key, byte[] value) throws IOException, HaloDBException {\n if (key.length > Byte.MAX_VALUE) {\n throw new HaloDBException(\"key length cannot exceed \" + Byte.MAX_VALUE);\n }\n\n //TODO: more fine-grained locking is possible. \n writeLock.lock();\n try {\n Record record = new Record(key, value);\n record.setSequenceNumber(getNextSequenceNumber());\n record.setVersion(Versions.CURRENT_DATA_FILE_VERSION);\n InMemoryIndexMetaData entry = writeRecordToFile(record);\n markPreviousVersionAsStale(key);\n\n //TODO: implement getAndSet and use the return value for\n //TODO: markPreviousVersionAsStale method.\n return inMemoryIndex.put(key, entry);\n } finally {\n writeLock.unlock();\n }\n }\n\n byte[] get(byte[] key, int attemptNumber) throws IOException, HaloDBException {\n if (attemptNumber > maxReadAttempts) {\n logger.error(\"Tried {} attempts but read failed\", attemptNumber-1);\n throw new HaloDBException(\"Tried \" + (attemptNumber-1) + \" attempts but failed.\");\n }\n InMemoryIndexMetaData metaData = inMemoryIndex.get(key);\n if (metaData == null) {\n return null;\n }\n\n HaloDBFile readFile = readFileMap.get(metaData.getFileId());\n if (readFile == null) {\n logger.debug(\"File {} not present. Compaction job would have deleted it. Retrying ...\", metaData.getFileId());\n return get(key, attemptNumber+1);\n }\n\n try {\n return readFile.readFromFile(metaData.getValueOffset(), metaData.getValueSize());\n }\n catch (ClosedChannelException e) {\n if (!isClosing) {\n logger.debug(\"File {} was closed. Compaction job would have deleted it. Retrying ...\", metaData.getFileId());\n return get(key, attemptNumber+1);\n }\n\n // trying to read after HaloDB.close() method called. \n throw e;\n }\n }\n\n int get(byte[] key, ByteBuffer buffer) throws IOException {\n InMemoryIndexMetaData metaData = inMemoryIndex.get(key);\n if (metaData == null) {\n return 0;\n }\n\n HaloDBFile readFile = readFileMap.get(metaData.getFileId());\n if (readFile == null) {\n logger.debug(\"File {} not present. Compaction job would have deleted it. Retrying ...\", metaData.getFileId());\n return get(key, buffer);\n }\n\n buffer.clear();\n buffer.limit(metaData.getValueSize());\n\n try {\n int read = readFile.readFromFile(metaData.getValueOffset(), buffer);\n buffer.flip();\n return read;\n }\n catch (ClosedChannelException e) {\n if (!isClosing) {\n logger.debug(\"File {} was closed. Compaction job would have deleted it. Retrying ...\", metaData.getFileId());\n return get(key, buffer);\n }\n\n // trying to read after HaloDB.close() method called.\n throw e;\n }\n }\n\n //TODO: use fine-grained lock if possible\n synchronized boolean takeSnapshot() {\n logger.info(\"Start generating the snapshot\");\n\n if (isTombstoneFilesMerging) {\n logger.info(\"DB is merging the tombstone files now. Wait it finished\");\n try {\n tombstoneMergeThread.join();\n } catch (InterruptedException e) {\n logger.error(\"Interrupted when waiting the tombstone files merging\");\n return false;\n }\n }\n\n try {\n final int currentWriteFileId;\n compactionManager.pauseCompactionThread();\n\n // Only support one snapshot now\n // TODO: support multiple snapshots if needed\n File snapshotDir = getSnapshotDirectory();\n if (snapshotDir.exists()) {\n logger.warn(\"The snapshot dir is already existed. Delete the old one.\");\n FileUtils.deleteDirectory(snapshotDir);\n }\n\n FileUtils.createDirectoryIfNotExists(snapshotDir);\n logger.info(\"Created directory for snapshot {}\", snapshotDir.toString());\n\n writeLock.lock();\n try {\n forceRollOverCurrentWriteFile();\n currentTombstoneFile = forceRollOverTombstoneFile(currentTombstoneFile);\n\n currentWriteFileId = currentWriteFile.getFileId();\n } catch (IOException e) {\n logger.warn(\"IO exception when rollover current write files\", e);\n\n return false;\n } finally {\n writeLock.unlock();\n }\n\n File[] filesToLink = dbDirectory.getPath().toFile()\n .listFiles(file -> {\n Matcher m = Constants.STORAGE_FILE_PATTERN.matcher(file.getName());\n return m.matches() && (Integer.parseInt(m.group(1)) < currentWriteFileId);\n });\n\n compactionManager.forceRolloverCurrentWriteFile();\n\n logger.info(\"Storage files number need to be linked: {}\", filesToLink.length);\n for (File file : filesToLink) {\n Path dest = Paths.get(snapshotDir.getAbsolutePath(), file.getName());\n logger.debug(\"Create file link from file {} to {}\", file.getName(),\n dest.toFile().getAbsoluteFile());\n Files.createLink(dest, file.toPath());\n }\n } catch(IOException e) {\n logger.warn(\"IOException when creating snapshot\", e);\n return false;\n } finally {\n compactionManager.resumeCompaction();\n }\n\n return true;\n }\n\n File getSnapshotDirectory() {\n Path dbDirectoryPath = dbDirectory.getPath();\n return Paths.get(dbDirectoryPath.toFile().getAbsolutePath(), SNAPSHOT_SUBDIR).toFile();\n }\n\n boolean clearSnapshot() {\n File snapshotDir = getSnapshotDirectory();\n if (snapshotDir.exists()) {\n try {\n FileUtils.deleteDirectory(snapshotDir);\n } catch (IOException e) {\n logger.error(\"snapshot deletion error\", e);\n return false;\n }\n\n return true;\n } else {\n logger.info(\"snapshot not existed\");\n return true;\n }\n }\n\n void delete(byte[] key) throws IOException {\n writeLock.lock();\n try {\n InMemoryIndexMetaData metaData = inMemoryIndex.get(key);\n if (metaData != null) {\n //TODO: implement a getAndRemove method in InMemoryIndex.\n inMemoryIndex.remove(key);\n TombstoneEntry entry =\n new TombstoneEntry(key, getNextSequenceNumber(), -1, Versions.CURRENT_TOMBSTONE_FILE_VERSION);\n currentTombstoneFile = rollOverTombstoneFile(entry, currentTombstoneFile);\n currentTombstoneFile.write(entry);\n markPreviousVersionAsStale(key, metaData);\n }\n } finally {\n writeLock.unlock();\n }\n }\n\n long size() {\n return inMemoryIndex.size();\n }\n\n void setIOErrorFlag() throws IOException {\n DBMetaData metaData = new DBMetaData(dbDirectory);\n metaData.loadFromFileIfExists();\n metaData.setIOError(true);\n metaData.storeToFile();\n }\n\n void pauseCompaction() throws IOException {\n compactionManager.pauseCompactionThread();\n }\n\n void resumeCompaction() {\n compactionManager.resumeCompaction();\n }\n\n private InMemoryIndexMetaData writeRecordToFile(Record record) throws IOException, HaloDBException {\n rollOverCurrentWriteFile(record);\n return currentWriteFile.writeRecord(record);\n }\n\n private void rollOverCurrentWriteFile(Record record) throws IOException {\n int size = record.getKey().length + record.getValue().length + Record.Header.HEADER_SIZE;\n if ((currentWriteFile == null || currentWriteFile.getWriteOffset() + size > options.getMaxFileSize())\n && !isClosing) {\n forceRollOverCurrentWriteFile();\n }\n }\n\n private void forceRollOverCurrentWriteFile() throws IOException {\n if (currentWriteFile != null) {\n currentWriteFile.flushToDisk();\n currentWriteFile.getIndexFile().flushToDisk();\n }\n currentWriteFile = createHaloDBFile(HaloDBFile.FileType.DATA_FILE);\n dbDirectory.syncMetaData();\n }\n\n private TombstoneFile rollOverTombstoneFile(TombstoneEntry entry, TombstoneFile tombstoneFile) throws IOException {\n int size = entry.getKey().length + TombstoneEntry.TOMBSTONE_ENTRY_HEADER_SIZE;\n if ((tombstoneFile == null ||\n tombstoneFile.getWriteOffset() + size > options.getMaxTombstoneFileSize()) && !isClosing) {\n tombstoneFile = forceRollOverTombstoneFile(tombstoneFile);\n }\n\n return tombstoneFile;\n }\n\n private TombstoneFile forceRollOverTombstoneFile(TombstoneFile tombstoneFile) throws IOException {\n if (tombstoneFile != null) {\n tombstoneFile.flushToDisk();\n tombstoneFile.close();\n }\n tombstoneFile = TombstoneFile.create(dbDirectory, getNextFileId(), options);\n dbDirectory.syncMetaData();\n\n return tombstoneFile;\n }\n\n\n private void markPreviousVersionAsStale(byte[] key) {\n InMemoryIndexMetaData recordMetaData = inMemoryIndex.get(key);\n if (recordMetaData != null) {\n markPreviousVersionAsStale(key, recordMetaData);\n }\n }\n\n private void markPreviousVersionAsStale(byte[] key, InMemoryIndexMetaData recordMetaData) {\n int staleRecordSize = Utils.getRecordSize(key.length, recordMetaData.getValueSize());\n addFileToCompactionQueueIfThresholdCrossed(recordMetaData.getFileId(), staleRecordSize);\n }\n\n void addFileToCompactionQueueIfThresholdCrossed(int fileId, int staleRecordSize) {\n HaloDBFile file = readFileMap.get(fileId);\n if (file == null)\n return;\n\n int staleSizeInFile = updateStaleDataMap(fileId, staleRecordSize);\n if (staleSizeInFile >= file.getSize() * options.getCompactionThresholdPerFile()) {\n\n // We don't want to compact the files the writer thread and the compaction thread is currently writing to.\n if (getCurrentWriteFileId() != fileId && compactionManager.getCurrentWriteFileId() != fileId) {\n if(compactionManager.submitFileForCompaction(fileId)) {\n staleDataPerFileMap.remove(fileId);\n }\n }\n }\n }\n\n private int updateStaleDataMap(int fileId, int staleDataSize) {\n return staleDataPerFileMap.merge(fileId, staleDataSize, (oldValue, newValue) -> oldValue + newValue);\n }\n\n void markFileAsCompacted(int fileId) {\n staleDataPerFileMap.remove(fileId);\n }\n\n InMemoryIndex getInMemoryIndex() {\n return inMemoryIndex;\n }\n\n HaloDBFile createHaloDBFile(HaloDBFile.FileType fileType) throws IOException {\n HaloDBFile file = HaloDBFile.create(dbDirectory, getNextFileId(), options, fileType);\n if(readFileMap.putIfAbsent(file.getFileId(), file) != null) {\n throw new IOException(\"Error while trying to create file \" + file.getName() + \" file with the given id already exists in the map\");\n }\n return file;\n }\n\n private List openDataFilesForReading() throws IOException {\n File[] files = dbDirectory.listDataFiles();\n\n List result = new ArrayList<>();\n for (File f : files) {\n HaloDBFile.FileType fileType = HaloDBFile.findFileType(f);\n result.add(HaloDBFile.openForReading(dbDirectory, f, fileType, options));\n }\n\n return result;\n }\n\n /**\n * Opens data files for reading and creates a map with file id as the key.\n * Also returns the latest file id in the directory which is then used\n * to determine the next file id.\n */\n private int buildReadFileMap() throws HaloDBException, IOException {\n int maxFileId = Integer.MIN_VALUE;\n\n for (HaloDBFile file : openDataFilesForReading()) {\n if (readFileMap.putIfAbsent(file.getFileId(), file) != null) {\n // There should only be a single file with a given file id.\n throw new HaloDBException(\"Found duplicate file with id \" + file.getFileId());\n }\n maxFileId = Math.max(maxFileId, file.getFileId());\n }\n\n if (maxFileId == Integer.MIN_VALUE) {\n // no files in the directory. use the current time as the first file id.\n maxFileId = Ints.checkedCast(System.currentTimeMillis() / 1000);\n }\n return maxFileId;\n }\n\n private int getNextFileId() {\n return nextFileId.incrementAndGet();\n }\n\n private Optional getLatestDataFile(HaloDBFile.FileType fileType) {\n return readFileMap.values()\n .stream()\n .filter(f -> f.getFileType() == fileType)\n .max(Comparator.comparingInt(HaloDBFile::getFileId));\n }\n\n private long buildInMemoryIndex() throws IOException {\n\n int nThreads = options.getBuildIndexThreads();\n logger.info(\"Building index in parallel with {} threads\", nThreads);\n\n ExecutorService executor = Executors.newFixedThreadPool(nThreads);\n try {\n return buildInMemoryIndex(executor);\n } finally {\n executor.shutdown();\n }\n }\n\n private long buildInMemoryIndex(ExecutorService executor) throws IOException {\n\n List indexFiles = dbDirectory.listIndexFiles();\n\n logger.info(\"About to scan {} index files to construct index ...\", indexFiles.size());\n\n long start = System.currentTimeMillis();\n long maxSequenceNumber = -1l;\n\n List indexFileTasks = new ArrayList<>();\n for (int fileId : indexFiles) {\n IndexFile indexFile = new IndexFile(fileId, dbDirectory, options);\n indexFileTasks.add(new ProcessIndexFileTask(indexFile, fileId));\n }\n\n try {\n List> results = executor.invokeAll(indexFileTasks);\n for (Future result : results) {\n maxSequenceNumber = Long.max(result.get(), maxSequenceNumber);\n }\n } catch (InterruptedException ie) {\n throw new IOException(\"Building index is interrupted\");\n } catch (ExecutionException ee) {\n throw new IOException(\"Error happened during building in-memory index\", ee);\n }\n logger.info(\"Completed scanning all index files in {}s\", (System.currentTimeMillis() - start) / 1000);\n\n // Scan all the tombstone files and remove records from index.\n start = System.currentTimeMillis();\n File[] tombStoneFiles = dbDirectory.listTombstoneFiles();\n logger.info(\"About to scan {} tombstone files ...\", tombStoneFiles.length);\n List tombstoneFileTasks = new ArrayList<>();\n for (File file : tombStoneFiles) {\n TombstoneFile tombstoneFile = new TombstoneFile(file, options, dbDirectory);\n tombstoneFileTasks.add(new ProcessTombstoneFileTask(tombstoneFile));\n }\n\n try {\n List> results = executor.invokeAll(tombstoneFileTasks);\n for (Future result : results) {\n maxSequenceNumber = Long.max(result.get(), maxSequenceNumber);\n }\n } catch (InterruptedException ie) {\n throw new IOException(\"Building index is interrupted\");\n } catch (ExecutionException ee) {\n throw new IOException(\"Error happened during building in-memory index\", ee);\n }\n logger.info(\"Completed scanning all tombstone files in {}s\", (System.currentTimeMillis() - start) / 1000);\n\n return maxSequenceNumber;\n }\n\n class ProcessIndexFileTask implements Callable {\n private final IndexFile indexFile;\n private final int fileId;\n\n public ProcessIndexFileTask(IndexFile indexFile, int fileId) {\n this.indexFile = indexFile;\n this.fileId = fileId;\n }\n\n @Override\n public Long call() throws IOException {\n long maxSequenceNumber = -1;\n indexFile.open();\n IndexFile.IndexFileIterator iterator = indexFile.newIterator();\n\n // build the in-memory index by scanning all index files.\n int count = 0, inserted = 0;\n while (iterator.hasNext()) {\n IndexFileEntry indexFileEntry = iterator.next();\n byte[] key = indexFileEntry.getKey();\n int recordOffset = indexFileEntry.getRecordOffset();\n int recordSize = indexFileEntry.getRecordSize();\n long sequenceNumber = indexFileEntry.getSequenceNumber();\n maxSequenceNumber = Long.max(sequenceNumber, maxSequenceNumber);\n int valueOffset = Utils.getValueOffset(recordOffset, key);\n int valueSize = recordSize - (Record.Header.HEADER_SIZE + key.length);\n count++;\n\n InMemoryIndexMetaData metaData = new InMemoryIndexMetaData(fileId, valueOffset, valueSize, sequenceNumber);\n\n if (!inMemoryIndex.putIfAbsent(key, metaData)) {\n while (true) {\n InMemoryIndexMetaData existing = inMemoryIndex.get(key);\n if (existing.getSequenceNumber() >= sequenceNumber) {\n // stale data, update stale data map.\n addFileToCompactionQueueIfThresholdCrossed(fileId, recordSize);\n break;\n }\n if (inMemoryIndex.replace(key, existing, metaData)) {\n // update stale data map for the previous version.\n addFileToCompactionQueueIfThresholdCrossed(existing.getFileId(), Utils.getRecordSize(key.length, existing.getValueSize()));\n inserted++;\n break;\n }\n }\n } else {\n inserted++;\n }\n }\n logger.debug(\"Completed scanning index file {}. Found {} records, inserted {} records\", fileId, count, inserted);\n indexFile.close();\n\n return maxSequenceNumber;\n }\n }\n\n class ProcessTombstoneFileTask implements Callable {\n private final TombstoneFile tombstoneFile;\n\n public ProcessTombstoneFileTask(TombstoneFile tombstoneFile) {\n this.tombstoneFile = tombstoneFile;\n }\n\n @Override\n public Long call() throws IOException {\n long maxSequenceNumber = -1;\n tombstoneFile.open();\n\n TombstoneFile rolloverFile = null;\n\n TombstoneFile.TombstoneFileIterator iterator = tombstoneFile.newIterator();\n\n long count = 0, active = 0, copied = 0;\n while (iterator.hasNext()) {\n TombstoneEntry entry = iterator.next();\n byte[] key = entry.getKey();\n long sequenceNumber = entry.getSequenceNumber();\n maxSequenceNumber = Long.max(sequenceNumber, maxSequenceNumber);\n count++;\n\n InMemoryIndexMetaData existing = inMemoryIndex.get(key);\n if (existing != null && existing.getSequenceNumber() < sequenceNumber) {\n // Found a tombstone record which happened after the version currently in index; remove.\n inMemoryIndex.remove(key);\n\n // update stale data map for the previous version.\n addFileToCompactionQueueIfThresholdCrossed(\n existing.getFileId(), Utils.getRecordSize(key.length, existing.getValueSize()));\n active++;\n\n if (options.isCleanUpTombstonesDuringOpen()) {\n rolloverFile = rollOverTombstoneFile(entry, rolloverFile);\n rolloverFile.write(entry);\n copied++;\n }\n }\n }\n logger.debug(\"Completed scanning tombstone file {}. Found {} tombstones, {} are still active\",\n tombstoneFile.getName(), count, active);\n tombstoneFile.close();\n\n if (options.isCleanUpTombstonesDuringOpen()) {\n logger.debug(\"Copied {} out of {} tombstones. Deleting {}\", copied, count, tombstoneFile.getName());\n if (rolloverFile != null) {\n logger.debug(\"Closing rollover tombstone file {}\", rolloverFile.getName());\n rolloverFile.flushToDisk();\n rolloverFile.close();\n }\n tombstoneFile.delete();\n }\n noOfTombstonesCopiedDuringOpen.addAndGet(copied);\n noOfTombstonesFoundDuringOpen.addAndGet(count);\n\n return maxSequenceNumber;\n }\n }\n\n HaloDBFile getHaloDBFile(int fileId) {\n return readFileMap.get(fileId);\n }\n\n void deleteHaloDBFile(int fileId) throws IOException {\n HaloDBFile file = readFileMap.get(fileId);\n\n if (file != null) {\n readFileMap.remove(fileId);\n file.delete();\n }\n\n staleDataPerFileMap.remove(fileId);\n }\n\n /**\n * If options.isCleanUpTombstonesDuringOpen set to true, all inactive entries,\n * i.e. physically deleted records, will be dropped during db open.\n * Refer to ProcessTombstoneFileTask class and buildInMemoryIndex()\n * To shorten db open time, active entries, i.e. not physically deleted\n * records, in each tombstone file are rolled over to a corresponding\n * new tombstone file. Therefore, the new tombstone file size might be very\n * small depends on number of active entries in each tombstone file.\n * A tombstone file won't be deleted as long as it has at least 1 active\n * entry. This function provide a way to merge small tombstone files in\n * offline mode. options.maxTombstoneFileSize still apply to merged file\n */\n private void mergeTombstoneFiles() {\n File[] tombStoneFiles = dbDirectory.listTombstoneFiles();\n\n logger.info(\"About to merge {} tombstone files ...\", tombStoneFiles.length);\n TombstoneFile mergedTombstoneFile = null;\n\n // Use compaction job rate as write rate limiter to avoid IO impact\n final RateLimiter rateLimiter = RateLimiter.create(options.getCompactionJobRate());\n\n for (File file : tombStoneFiles) {\n TombstoneFile tombstoneFile = new TombstoneFile(file, options, dbDirectory);\n if (currentTombstoneFile != null && tombstoneFile.getName().equals(currentTombstoneFile.getName())) {\n continue; // not touch current tombstone file\n }\n\n try {\n tombstoneFile.open();\n TombstoneFile.TombstoneFileIterator iterator = tombstoneFile.newIterator();\n\n long count = 0;\n while (iterator.hasNext()) {\n TombstoneEntry entry = iterator.next();\n rateLimiter.acquire(entry.size());\n count++;\n mergedTombstoneFile = rollOverTombstoneFile(entry, mergedTombstoneFile);\n mergedTombstoneFile.write(entry);\n }\n if (count > 0) {\n logger.debug(\"Merged {} tombstones from {} to {}\",\n count, tombstoneFile.getName(), mergedTombstoneFile.getName());\n }\n tombstoneFile.close();\n tombstoneFile.delete();\n } catch (IOException e) {\n logger.error(\"IO exception when merging tombstone file\", e);\n }\n }\n\n if (mergedTombstoneFile != null) {\n try {\n mergedTombstoneFile.close();\n } catch (IOException e) {\n logger.error(\"IO exception when closing tombstone file: {}\", mergedTombstoneFile.getName(), e);\n }\n }\n logger.info(\"Tombstone files count, before merge:{}, after merge:{}\",\n tombStoneFiles.length, dbDirectory.listTombstoneFiles().length);\n isTombstoneFilesMerging = false;\n }\n\n private void repairFiles() {\n getLatestDataFile(HaloDBFile.FileType.DATA_FILE).ifPresent(file -> {\n try {\n logger.info(\"Repairing file {}.data\", file.getFileId());\n HaloDBFile repairedFile = file.repairFile(dbDirectory);\n readFileMap.put(repairedFile.getFileId(), repairedFile);\n }\n catch (IOException e) {\n throw new RuntimeException(\"Exception while repairing data file \" + file.getFileId() + \" which might be corrupted\", e);\n }\n });\n getLatestDataFile(HaloDBFile.FileType.COMPACTED_FILE).ifPresent(file -> {\n try {\n logger.info(\"Repairing file {}.datac\", file.getFileId());\n HaloDBFile repairedFile = file.repairFile(dbDirectory);\n readFileMap.put(repairedFile.getFileId(), repairedFile);\n }\n catch (IOException e) {\n throw new RuntimeException(\"Exception while repairing datac file \" + file.getFileId() + \" which might be corrupted\", e);\n }\n });\n\n File[] tombstoneFiles = dbDirectory.listTombstoneFiles();\n if (tombstoneFiles != null && tombstoneFiles.length > 0) {\n TombstoneFile lastFile = new TombstoneFile(tombstoneFiles[tombstoneFiles.length-1], options, dbDirectory);\n try {\n logger.info(\"Repairing {} file\", lastFile.getName());\n lastFile.open();\n TombstoneFile repairedFile = lastFile.repairFile(dbDirectory);\n repairedFile.close();\n } catch (IOException e) {\n throw new RuntimeException(\"Exception while repairing tombstone file \" + lastFile.getName() + \" which might be corrupted\", e);\n }\n }\n }\n\n private FileLock getLock() throws HaloDBException {\n try {\n FileLock lock = FileChannel.open(dbDirectory.getPath().resolve(\"LOCK\"), StandardOpenOption.CREATE, StandardOpenOption.WRITE).tryLock();\n if (lock == null) {\n logger.error(\"Error while opening db. Another process already holds a lock to this db.\");\n throw new HaloDBException(\"Another process already holds a lock for this db.\");\n }\n\n return lock;\n }\n catch (OverlappingFileLockException e) {\n logger.error(\"Error while opening db. Another process already holds a lock to this db.\");\n throw new HaloDBException(\"Another process already holds a lock for this db.\");\n }\n catch (IOException e) {\n logger.error(\"Error while trying to get a lock on the db.\", e);\n throw new HaloDBException(\"Error while trying to get a lock on the db.\", e);\n }\n }\n\n DBDirectory getDbDirectory() {\n return dbDirectory;\n }\n\n Set listDataFileIds() {\n return new HashSet<>(readFileMap.keySet());\n }\n\n boolean isRecordFresh(byte[] key, InMemoryIndexMetaData metaData) {\n InMemoryIndexMetaData currentMeta = inMemoryIndex.get(key);\n\n return\n currentMeta != null\n &&\n metaData.getFileId() == currentMeta.getFileId()\n &&\n metaData.getValueOffset() == currentMeta.getValueOffset();\n }\n\n private long getNextSequenceNumber() {\n return nextSequenceNumber++;\n }\n\n private int getCurrentWriteFileId() {\n return currentWriteFile != null ? currentWriteFile.getFileId() : -1;\n }\n\n private static void checkIfOptionsAreCorrect(HaloDBOptions options) {\n if (options.isUseMemoryPool() && (options.getFixedKeySize() < 0 || options.getFixedKeySize() > Byte.MAX_VALUE)) {\n throw new IllegalArgumentException(\"fixedKeySize must be set and should be less than 128 when using memory pool\");\n }\n }\n\n boolean isClosing() {\n return isClosing;\n }\n\n HaloDBStats stats() {\n OffHeapHashTableStats stats = inMemoryIndex.stats();\n return new HaloDBStats(\n statsResetTime,\n stats.getSize(),\n compactionManager.isCompactionRunning(),\n compactionManager.noOfFilesPendingCompaction(),\n computeStaleDataMapForStats(),\n stats.getRehashCount(),\n inMemoryIndex.getNoOfSegments(),\n inMemoryIndex.getMaxSizeOfEachSegment(),\n stats.getSegmentStats(),\n dbDirectory.listDataFiles().length,\n dbDirectory.listTombstoneFiles().length,\n noOfTombstonesFoundDuringOpen.get(),\n options.isCleanUpTombstonesDuringOpen() ?\n noOfTombstonesFoundDuringOpen.get() - noOfTombstonesCopiedDuringOpen.get() : 0,\n compactionManager.getNumberOfRecordsCopied(),\n compactionManager.getNumberOfRecordsReplaced(),\n compactionManager.getNumberOfRecordsScanned(),\n compactionManager.getSizeOfRecordsCopied(),\n compactionManager.getSizeOfFilesDeleted(),\n compactionManager.getSizeOfFilesDeleted()-compactionManager.getSizeOfRecordsCopied(),\n compactionManager.getCompactionJobRateSinceBeginning(),\n options.clone()\n );\n }\n\n synchronized void resetStats() {\n inMemoryIndex.resetStats();\n compactionManager.resetStats();\n statsResetTime = System.currentTimeMillis();\n }\n\n private Map computeStaleDataMapForStats() {\n Map stats = new HashMap<>();\n staleDataPerFileMap.forEach((fileId, staleData) -> {\n HaloDBFile file = readFileMap.get(fileId);\n if (file != null && file.getSize() > 0) {\n double stalePercent = (1.0*staleData/file.getSize()) * 100;\n stats.put(fileId, stalePercent);\n }\n });\n\n return stats;\n }\n\n // Used only in tests.\n @VisibleForTesting\n boolean isCompactionComplete() {\n return compactionManager.isCompactionComplete();\n }\n\n @VisibleForTesting\n boolean isTombstoneFilesMerging() {\n return isTombstoneFilesMerging;\n }\n}\n" }, { "path": "src/main/java/com/oath/halodb/HaloDBIterator.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport java.io.IOException;\nimport java.nio.channels.ClosedChannelException;\nimport java.util.Iterator;\nimport java.util.NoSuchElementException;\n\npublic class HaloDBIterator implements Iterator {\n private static final Logger logger = LoggerFactory.getLogger(HaloDBIterator.class);\n\n private Iterator outer;\n private Iterator inner;\n private HaloDBFile currentFile;\n\n private Record next;\n\n private final HaloDBInternal dbInternal;\n\n HaloDBIterator(HaloDBInternal dbInternal) {\n this.dbInternal = dbInternal;\n outer = dbInternal.listDataFileIds().iterator();\n }\n\n @Override\n public boolean hasNext() {\n if (next != null) {\n return true;\n }\n\n try {\n // inner == null means this is the first time hasNext() is called.\n // use moveToNextFile() to move to the first file.\n if (inner == null && !moveToNextFile()) {\n return false;\n }\n\n do {\n if (readNextRecord()) {\n return true;\n }\n } while (moveToNextFile());\n\n return false;\n\n } catch (IOException e) {\n logger.error(\"Error in Iterator\", e);\n return false;\n }\n }\n\n @Override\n public Record next() {\n if (hasNext()) {\n Record record = next;\n next = null;\n return record;\n }\n throw new NoSuchElementException();\n }\n\n private boolean moveToNextFile() throws IOException {\n while (outer.hasNext()) {\n int fileId = outer.next();\n currentFile = dbInternal.getHaloDBFile(fileId);\n if (currentFile != null) {\n try {\n inner = currentFile.getIndexFile().newIterator();\n return true;\n } catch (ClosedChannelException e) {\n if (dbInternal.isClosing()) {\n //TODO: define custom Exception classes for HaloDB.\n throw new RuntimeException(\"DB is closing\");\n }\n logger.debug(\"Index file {} closed, probably by compaction thread. Skipping to next one\", fileId);\n }\n }\n logger.debug(\"Data file {} deleted, probably by compaction thread. Skipping to next one\", fileId);\n }\n\n return false;\n }\n\n private boolean readNextRecord() {\n while (inner.hasNext()) {\n IndexFileEntry entry = inner.next();\n try {\n try {\n next = readRecordFromDataFile(entry);\n if (next != null) {\n return true;\n }\n } catch (ClosedChannelException e) {\n if (dbInternal.isClosing()) {\n throw new RuntimeException(\"DB is closing\");\n }\n logger.debug(\"Data file {} closed, probably by compaction thread. Skipping to next one\", currentFile.getFileId());\n break;\n }\n } catch (IOException e) {\n logger.info(\"Error in iterator\", e);\n break;\n }\n }\n return false;\n }\n\n private Record readRecordFromDataFile(IndexFileEntry entry) throws IOException {\n InMemoryIndexMetaData meta = Utils.getMetaData(entry, currentFile.getFileId());\n Record record = null;\n if (dbInternal.isRecordFresh(entry.getKey(), meta)) {\n byte[] value = currentFile.readFromFile(\n Utils.getValueOffset(entry.getRecordOffset(), entry.getKey()),\n Utils.getValueSize(entry.getRecordSize(), entry.getKey()));\n record = new Record(entry.getKey(), value);\n record.setRecordMetaData(meta);\n }\n return record;\n }\n}\n" }, { "path": "src/main/java/com/oath/halodb/HaloDBKeyIterator.java", "content": "package com.oath.halodb;\n\nimport java.io.IOException;\nimport java.nio.channels.ClosedChannelException;\nimport java.util.*;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\npublic class HaloDBKeyIterator implements Iterator{\n private static final Logger logger = LoggerFactory.getLogger(HaloDBIterator.class);\n\n private Iterator outer;\n private Iterator inner;\n private HaloDBFile currentFile;\n\n private RecordKey next;\n\n private final HaloDBInternal dbInternal;\n\n HaloDBKeyIterator(HaloDBInternal dbInternal) {\n this.dbInternal = dbInternal;\n outer = dbInternal.listDataFileIds().iterator();\n }\n\n @Override\n public boolean hasNext() {\n if (next != null) {\n return true;\n }\n\n try {\n // inner == null means this is the first time hasNext() is called.\n // use moveToNextFile() to move to the first file.\n if (inner == null && !moveToNextFile()) {\n return false;\n }\n\n do {\n if (readNextRecord()) {\n return true;\n }\n } while (moveToNextFile());\n\n return false;\n\n } catch (IOException e) {\n logger.error(\"Error in Iterator\", e);\n return false;\n }\n }\n\n @Override\n public RecordKey next() {\n if (hasNext()) {\n RecordKey key = next;\n next = null;\n return key;\n }\n throw new NoSuchElementException();\n }\n\n private boolean moveToNextFile() throws IOException {\n while (outer.hasNext()) {\n int fileId = outer.next();\n currentFile = dbInternal.getHaloDBFile(fileId);\n if (currentFile != null) {\n try {\n inner = currentFile.getIndexFile().newIterator();\n return true;\n } catch (ClosedChannelException e) {\n if (dbInternal.isClosing()) {\n //TODO: define custom Exception classes for HaloDB.\n throw new RuntimeException(\"DB is closing\");\n }\n logger.debug(\"Index file {} closed, probably by compaction thread. Skipping to next one\", fileId);\n }\n }\n logger.debug(\"Data file {} deleted, probably by compaction thread. Skipping to next one\", fileId);\n }\n\n return false;\n }\n\n private boolean readNextRecord() {\n while (inner.hasNext()) {\n IndexFileEntry entry = inner.next();\n try {\n try {\n next = readValidRecordKey(entry);\n if (next != null) {\n return true;\n }\n } catch (ClosedChannelException e) {\n if (dbInternal.isClosing()) {\n throw new RuntimeException(\"DB is closing\");\n }\n logger.debug(\"Data file {} closed, probably by compaction thread. Skipping to next one\", currentFile.getFileId());\n break;\n }\n } catch (IOException e) {\n logger.info(\"Error in iterator\", e);\n break;\n }\n }\n return false;\n }\n\n private RecordKey readValidRecordKey(IndexFileEntry entry) throws IOException {\n InMemoryIndexMetaData meta = Utils.getMetaData(entry, currentFile.getFileId());\n RecordKey key = null;\n if (dbInternal.isRecordFresh(entry.getKey(), meta)) {\n key = new RecordKey(entry.getKey());\n }\n return key;\n }\n}\n" }, { "path": "src/main/java/com/oath/halodb/HaloDBOptions.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport com.google.common.base.MoreObjects;\n\npublic class HaloDBOptions implements Cloneable {\n\n // threshold of stale data at which file needs to be compacted.\n private double compactionThresholdPerFile = 0.75;\n\n private int maxFileSize = 1024 * 1024; /* 1mb file recordSize */\n\n // To keep backward compatibility, initialize to 0 which means\n // it will fall back to use maxFileSize, see the getter below\n private int maxTombstoneFileSize = 0;\n\n // Data will be flushed to disk after flushDataSizeBytes have been written.\n // -1 disables explicit flushing and let the kernel handle it.\n private long flushDataSizeBytes = -1;\n\n // Write call will sync data to disk before returning.\n // If enabled trades off write throughput for durability.\n private boolean syncWrite = false;\n\n private int numberOfRecords = 1_000_000;\n\n // MB of data to be compacted per second.\n private int compactionJobRate = 1024 * 1024 * 1024;\n\n private boolean cleanUpInMemoryIndexOnClose = false;\n\n private boolean cleanUpTombstonesDuringOpen = false;\n\n private boolean useMemoryPool = false;\n\n private int fixedKeySize = Byte.MAX_VALUE;\n\n private int memoryPoolChunkSize = 16 * 1024 * 1024;\n\n // Number of threads to scan index and tombstone files\n // to build in-memory index at db open\n private int buildIndexThreads = 1;\n\n // Just to avoid clients having to deal with CloneNotSupportedException\n public HaloDBOptions clone() {\n try {\n return (HaloDBOptions) super.clone();\n } catch (CloneNotSupportedException e) {\n return null;\n }\n }\n\n @Override\n public String toString() {\n return MoreObjects.toStringHelper(\"\")\n .add(\"compactionThresholdPerFile\", compactionThresholdPerFile)\n .add(\"maxFileSize\", maxFileSize)\n .add(\"maxTombstoneFileSize\", getMaxTombstoneFileSize())\n .add(\"flushDataSizeBytes\", flushDataSizeBytes)\n .add(\"syncWrite\", syncWrite)\n .add(\"numberOfRecords\", numberOfRecords)\n .add(\"compactionJobRate\", compactionJobRate)\n .add(\"cleanUpInMemoryIndexOnClose\", cleanUpInMemoryIndexOnClose)\n .add(\"cleanUpTombstonesDuringOpen\", cleanUpTombstonesDuringOpen)\n .add(\"useMemoryPool\", useMemoryPool)\n .add(\"fixedKeySize\", fixedKeySize)\n .add(\"memoryPoolChunkSize\", memoryPoolChunkSize)\n .add(\"buildIndexThreads\", buildIndexThreads)\n .toString();\n }\n\n public void setCompactionThresholdPerFile(double compactionThresholdPerFile) {\n this.compactionThresholdPerFile = compactionThresholdPerFile;\n }\n\n public void setMaxFileSize(int maxFileSize) {\n if (maxFileSize <= 0) {\n throw new IllegalArgumentException(\"maxFileSize should be > 0\");\n }\n this.maxFileSize = maxFileSize;\n }\n\n public void setMaxTombstoneFileSize(int maxFileSize) {\n if (maxFileSize <= 0) {\n throw new IllegalArgumentException(\"maxFileSize should be > 0\");\n }\n this.maxTombstoneFileSize = maxFileSize;\n }\n\n public void setFlushDataSizeBytes(long flushDataSizeBytes) {\n this.flushDataSizeBytes = flushDataSizeBytes;\n }\n\n public void setNumberOfRecords(int numberOfRecords) {\n this.numberOfRecords = numberOfRecords;\n }\n\n public void setCompactionJobRate(int compactionJobRate) {\n this.compactionJobRate = compactionJobRate;\n }\n\n public void setCleanUpInMemoryIndexOnClose(boolean cleanUpInMemoryIndexOnClose) {\n this.cleanUpInMemoryIndexOnClose = cleanUpInMemoryIndexOnClose;\n }\n\n public double getCompactionThresholdPerFile() {\n return compactionThresholdPerFile;\n }\n\n public int getMaxFileSize() {\n return maxFileSize;\n }\n\n public int getMaxTombstoneFileSize() {\n return maxTombstoneFileSize > 0 ? maxTombstoneFileSize : maxFileSize;\n }\n\n public long getFlushDataSizeBytes() {\n return flushDataSizeBytes;\n }\n\n public int getNumberOfRecords() {\n return numberOfRecords;\n }\n\n public int getCompactionJobRate() {\n return compactionJobRate;\n }\n\n public boolean isCleanUpInMemoryIndexOnClose() {\n return cleanUpInMemoryIndexOnClose;\n }\n\n public boolean isCleanUpTombstonesDuringOpen() {\n return cleanUpTombstonesDuringOpen;\n }\n\n public void setCleanUpTombstonesDuringOpen(boolean cleanUpTombstonesDuringOpen) {\n this.cleanUpTombstonesDuringOpen = cleanUpTombstonesDuringOpen;\n }\n \n public boolean isUseMemoryPool() {\n return useMemoryPool;\n }\n\n public void setUseMemoryPool(boolean useMemoryPool) {\n this.useMemoryPool = useMemoryPool;\n }\n\n public int getFixedKeySize() {\n return fixedKeySize;\n }\n\n public void setFixedKeySize(int fixedKeySize) {\n this.fixedKeySize = fixedKeySize;\n }\n\n public int getMemoryPoolChunkSize() {\n return memoryPoolChunkSize;\n }\n\n public void setMemoryPoolChunkSize(int memoryPoolChunkSize) {\n this.memoryPoolChunkSize = memoryPoolChunkSize;\n }\n\n public boolean isSyncWrite() {\n return syncWrite;\n }\n\n public void enableSyncWrites(boolean syncWrites) {\n this.syncWrite = syncWrites;\n }\n\n public int getBuildIndexThreads() {\n return buildIndexThreads;\n }\n\n public void setBuildIndexThreads(int buildIndexThreads) {\n int numOfProcessors = Runtime.getRuntime().availableProcessors();\n if (buildIndexThreads <= 0 || buildIndexThreads > numOfProcessors) {\n throw new IllegalArgumentException(\"buildIndexThreads should be > 0 and <= \" + numOfProcessors);\n }\n this.buildIndexThreads = buildIndexThreads;\n }\n\n // to be used only in tests.\n private boolean isCompactionDisabled = false;\n \n // not visible to outside the package.\n // to be used only in tests.\n void setCompactionDisabled(boolean compactionDisabled) {\n isCompactionDisabled = compactionDisabled;\n }\n boolean isCompactionDisabled() {\n return isCompactionDisabled;\n }\n}\n" }, { "path": "src/main/java/com/oath/halodb/HaloDBStats.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport com.google.common.base.MoreObjects;\n\nimport java.util.Arrays;\nimport java.util.HashMap;\nimport java.util.Map;\n\npublic class HaloDBStats {\n\n private final long statsResetTime;\n\n private final long size;\n\n private final int numberOfFilesPendingCompaction;\n private final Map staleDataPercentPerFile;\n\n private final long rehashCount;\n private final long numberOfSegments;\n private final long maxSizePerSegment;\n private final SegmentStats[] segmentStats;\n\n private final long numberOfTombstonesFoundDuringOpen;\n private final long numberOfTombstonesCleanedUpDuringOpen;\n\n private final int numberOfDataFiles;\n private final int numberOfTombstoneFiles;\n\n private final long numberOfRecordsCopied;\n private final long numberOfRecordsReplaced;\n private final long numberOfRecordsScanned;\n private final long sizeOfRecordsCopied;\n private final long sizeOfFilesDeleted;\n private final long sizeReclaimed;\n\n private final long compactionRateInInternal;\n private final long compactionRateSinceBeginning;\n\n private final boolean isCompactionRunning;\n\n private final HaloDBOptions options;\n\n public HaloDBStats(long statsResetTime, long size, boolean isCompactionRunning, int numberOfFilesPendingCompaction,\n Map staleDataPercentPerFile, long rehashCount, long numberOfSegments,\n long maxSizePerSegment, SegmentStats[] segmentStats,\n int numberOfDataFiles, int numberOfTombstoneFiles,\n long numberOfTombstonesFoundDuringOpen, long numberOfTombstonesCleanedUpDuringOpen,\n long numberOfRecordsCopied, long numberOfRecordsReplaced, long numberOfRecordsScanned,\n long sizeOfRecordsCopied, long sizeOfFilesDeleted, long sizeReclaimed,\n long compactionRateSinceBeginning, HaloDBOptions options) {\n this.statsResetTime = statsResetTime;\n this.size = size;\n this.numberOfFilesPendingCompaction = numberOfFilesPendingCompaction;\n this.staleDataPercentPerFile = staleDataPercentPerFile;\n this.rehashCount = rehashCount;\n this.numberOfSegments = numberOfSegments;\n this.maxSizePerSegment = maxSizePerSegment;\n this.segmentStats = segmentStats;\n this.numberOfDataFiles = numberOfDataFiles;\n this.numberOfTombstoneFiles = numberOfTombstoneFiles;\n this.numberOfTombstonesFoundDuringOpen = numberOfTombstonesFoundDuringOpen;\n this.numberOfTombstonesCleanedUpDuringOpen = numberOfTombstonesCleanedUpDuringOpen;\n this.numberOfRecordsCopied = numberOfRecordsCopied;\n this.numberOfRecordsReplaced = numberOfRecordsReplaced;\n this.numberOfRecordsScanned = numberOfRecordsScanned;\n this.sizeOfRecordsCopied = sizeOfRecordsCopied;\n this.sizeOfFilesDeleted = sizeOfFilesDeleted;\n this.sizeReclaimed = sizeReclaimed;\n this.compactionRateSinceBeginning = compactionRateSinceBeginning;\n this.isCompactionRunning = isCompactionRunning;\n\n long intervalTimeInSeconds = (System.currentTimeMillis() - statsResetTime)/1000;\n if (intervalTimeInSeconds > 0) {\n this.compactionRateInInternal = sizeOfRecordsCopied/intervalTimeInSeconds;\n }\n else {\n this.compactionRateInInternal = 0;\n }\n\n this.options = options;\n }\n\n public long getSize() {\n return size;\n }\n\n public int getNumberOfFilesPendingCompaction() {\n return numberOfFilesPendingCompaction;\n }\n\n public Map getStaleDataPercentPerFile() {\n return staleDataPercentPerFile;\n }\n\n public long getRehashCount() {\n return rehashCount;\n }\n\n public long getNumberOfSegments() {\n return numberOfSegments;\n }\n\n public long getMaxSizePerSegment() {\n return maxSizePerSegment;\n }\n\n public long getNumberOfRecordsCopied() {\n return numberOfRecordsCopied;\n }\n\n public long getNumberOfRecordsReplaced() {\n return numberOfRecordsReplaced;\n }\n\n public long getNumberOfRecordsScanned() {\n return numberOfRecordsScanned;\n }\n\n public long getSizeOfRecordsCopied() {\n return sizeOfRecordsCopied;\n }\n\n public long getSizeOfFilesDeleted() {\n return sizeOfFilesDeleted;\n }\n\n public long getSizeReclaimed() {\n return sizeReclaimed;\n }\n\n public HaloDBOptions getOptions() {\n return options;\n }\n\n public int getNumberOfDataFiles() {\n return numberOfDataFiles;\n }\n\n public int getNumberOfTombstoneFiles() {\n return numberOfTombstoneFiles;\n }\n\n public long getNumberOfTombstonesFoundDuringOpen() {\n return numberOfTombstonesFoundDuringOpen;\n }\n\n public long getNumberOfTombstonesCleanedUpDuringOpen() {\n return numberOfTombstonesCleanedUpDuringOpen;\n }\n\n public SegmentStats[] getSegmentStats() {\n return segmentStats;\n }\n\n public long getCompactionRateInInternal() {\n return compactionRateInInternal;\n }\n\n public long getCompactionRateSinceBeginning() {\n return compactionRateSinceBeginning;\n }\n\n public boolean isCompactionRunning() {\n return isCompactionRunning;\n }\n\n @Override\n public String toString() {\n return MoreObjects.toStringHelper(\"\")\n .add(\"statsResetTime\", statsResetTime)\n .add(\"size\", size)\n .add(\"Options\", options)\n .add(\"isCompactionRunning\", isCompactionRunning)\n .add(\"CompactionJobRateInInterval\", getUnit(compactionRateInInternal))\n .add(\"CompactionJobRateSinceBeginning\", getUnit(compactionRateSinceBeginning))\n .add(\"numberOfFilesPendingCompaction\", numberOfFilesPendingCompaction)\n .add(\"numberOfRecordsCopied\", numberOfRecordsCopied)\n .add(\"numberOfRecordsReplaced\", numberOfRecordsReplaced)\n .add(\"numberOfRecordsScanned\", numberOfRecordsScanned)\n .add(\"sizeOfRecordsCopied\", sizeOfRecordsCopied)\n .add(\"sizeOfFilesDeleted\", sizeOfFilesDeleted)\n .add(\"sizeReclaimed\", sizeReclaimed)\n .add(\"rehashCount\", rehashCount)\n .add(\"maxSizePerSegment\", maxSizePerSegment)\n .add(\"numberOfDataFiles\", numberOfDataFiles)\n .add(\"numberOfTombstoneFiles\", numberOfTombstoneFiles)\n .add(\"numberOfTombstonesFoundDuringOpen\", numberOfTombstonesFoundDuringOpen)\n .add(\"numberOfTombstonesCleanedUpDuringOpen\", numberOfTombstonesCleanedUpDuringOpen)\n .add(\"segmentStats\", Arrays.toString(segmentStats))\n .add(\"numberOfSegments\", numberOfSegments)\n .add(\"staleDataPercentPerFile\", staleDataMapToString())\n .toString();\n }\n\n public Map toStringMap() {\n Map map = new HashMap<>();\n map.put(\"statsResetTime\", String.valueOf(statsResetTime));\n map.put(\"size\", String.valueOf(size));\n map.put(\"Options\", String.valueOf(options));\n map.put(\"isCompactionRunning\", String.valueOf(isCompactionRunning));\n map.put(\"CompactionJobRateInInterval\", String.valueOf(getUnit(compactionRateInInternal)));\n map.put(\"CompactionJobRateSinceBeginning\", String.valueOf(getUnit(compactionRateSinceBeginning)));\n map.put(\"numberOfFilesPendingCompaction\", String.valueOf(numberOfFilesPendingCompaction));\n map.put(\"numberOfRecordsCopied\", String.valueOf(numberOfRecordsCopied));\n map.put(\"numberOfRecordsReplaced\", String.valueOf(numberOfRecordsReplaced));\n map.put(\"numberOfRecordsScanned\", String.valueOf(numberOfRecordsScanned));\n map.put(\"sizeOfRecordsCopied\", String.valueOf(sizeOfRecordsCopied));\n map.put(\"sizeOfFilesDeleted\", String.valueOf(sizeOfFilesDeleted));\n map.put(\"sizeReclaimed\", String.valueOf(sizeReclaimed));\n map.put(\"rehashCount\", String.valueOf(rehashCount));\n map.put(\"maxSizePerSegment\", String.valueOf(maxSizePerSegment));\n map.put(\"numberOfDataFiles\", String.valueOf(numberOfDataFiles));\n map.put(\"numberOfTombstoneFiles\", String.valueOf(numberOfTombstoneFiles));\n map.put(\"numberOfTombstonesFoundDuringOpen\", String.valueOf(numberOfTombstonesFoundDuringOpen));\n map.put(\"numberOfTombstonesCleanedUpDuringOpen\", String.valueOf(numberOfTombstonesCleanedUpDuringOpen));\n map.put(\"segmentStats\", String.valueOf(Arrays.toString(segmentStats)));\n map.put(\"numberOfSegments\", String.valueOf(numberOfSegments));\n map.put(\"staleDataPercentPerFile\", String.valueOf(staleDataMapToString()));\n\n return map;\n }\n\n private String staleDataMapToString() {\n StringBuilder builder = new StringBuilder(\"[\");\n boolean isFirst = true;\n\n for (Map.Entry e : staleDataPercentPerFile.entrySet()) {\n if (!isFirst) {\n builder.append(\", \");\n }\n isFirst = false;\n builder.append(\"{\");\n builder.append(e.getKey());\n builder.append(\"=\");\n builder.append(String.format(\"%.1f\", e.getValue()));\n builder.append(\"}\");\n }\n builder.append(\"]\");\n return builder.toString();\n }\n\n private static final String gbRateUnit = \" GB/second\";\n private static final String mbRateUnit = \" MB/second\";\n private static final String kbRateUnit = \" KB/second\";\n\n private static final long GB = 1024 * 1024 * 1024;\n private static final long MB = 1024 * 1024;\n private static final long KB = 1024;\n\n private String getUnit(long value) {\n long temp = value / GB;\n if (temp >= 1) {\n return temp + gbRateUnit;\n }\n\n temp = value / MB;\n if (temp >= 1) {\n return temp + mbRateUnit;\n }\n\n temp = value / KB;\n return temp + kbRateUnit;\n }\n\n}\n" }, { "path": "src/main/java/com/oath/halodb/HashAlgorithm.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\n// This code is a derivative work heavily modified from the OHC project. See NOTICE file for copyright and license.\n\npackage com.oath.halodb;\n\nenum HashAlgorithm {\n MURMUR3,\n\n CRC32,\n\n XX\n}\n" }, { "path": "src/main/java/com/oath/halodb/HashTableUtil.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\n// This code is a derivative work heavily modified from the OHC project. See NOTICE file for copyright and license.\n\npackage com.oath.halodb;\n\nfinal class HashTableUtil {\n\n// Hash bucket-table\n\n static final long NON_MEMORY_POOL_BUCKET_ENTRY_LEN = 8;\n static final long MEMORY_POOL_BUCKET_ENTRY_LEN = 5;\n\n static long allocLen(long keyLen, long valueLen) {\n return NonMemoryPoolHashEntries.ENTRY_OFF_DATA + keyLen + valueLen;\n }\n\n static int bitNum(long val) {\n int bit = 0;\n for (; val != 0L; bit++) {\n val >>>= 1;\n }\n return bit;\n }\n\n static long roundUpToPowerOf2(long number, long max) {\n return number >= max\n ? max\n : (number > 1) ? Long.highestOneBit((number - 1) << 1) : 1;\n }\n}\n" }, { "path": "src/main/java/com/oath/halodb/HashTableValueSerializer.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\n// This code is a derivative work heavily modified from the OHC project. See NOTICE file for copyright and license.\n\npackage com.oath.halodb;\n\nimport java.nio.ByteBuffer;\n\n/**\n * Serialize and deserialize cached data using {@link ByteBuffer}\n */\ninterface HashTableValueSerializer {\n\n void serialize(T value, ByteBuffer buf);\n\n T deserialize(ByteBuffer buf);\n\n int serializedSize(T value);\n}\n\n" }, { "path": "src/main/java/com/oath/halodb/Hasher.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\n// This code is a derivative work heavily modified from the OHC project. See NOTICE file for copyright and license.\n\npackage com.oath.halodb;\n\nimport net.jpountz.xxhash.XXHashFactory;\n\nimport java.util.zip.CRC32;\n\nabstract class Hasher {\n\n static Hasher create(HashAlgorithm hashAlgorithm) {\n String cls = forAlg(hashAlgorithm);\n try {\n return (Hasher) Class.forName(cls).newInstance();\n } catch (ClassNotFoundException e) {\n if (hashAlgorithm == HashAlgorithm.XX) {\n cls = forAlg(HashAlgorithm.CRC32);\n try {\n return (Hasher) Class.forName(cls).newInstance();\n } catch (InstantiationException | ClassNotFoundException | IllegalAccessException e1) {\n throw new RuntimeException(e1);\n }\n }\n throw new RuntimeException(e);\n } catch (InstantiationException | IllegalAccessException e) {\n throw new RuntimeException(e);\n }\n }\n\n private static String forAlg(HashAlgorithm hashAlgorithm) {\n return Hasher.class.getName()\n + '$'\n + hashAlgorithm.name().substring(0, 1)\n + hashAlgorithm.name().substring(1).toLowerCase()\n + \"Hash\";\n }\n\n abstract long hash(byte[] array);\n\n abstract long hash(long address, long offset, int length);\n\n static final class Crc32Hash extends Hasher {\n\n long hash(byte[] array) {\n CRC32 crc = new CRC32();\n crc.update(array);\n long h = crc.getValue();\n h |= h << 32;\n return h;\n }\n\n long hash(long address, long offset, int length) {\n return Uns.crc32(address, offset, length);\n }\n }\n\n static final class Murmur3Hash extends Hasher {\n\n long hash(byte[] array) {\n int o = 0;\n int r = array.length;\n\n long h1 = 0L;\n long h2 = 0L;\n long k1, k2;\n\n for (; r >= 16; r -= 16) {\n k1 = getLong(array, o);\n o += 8;\n k2 = getLong(array, o);\n o += 8;\n\n // bmix64()\n\n h1 ^= mixK1(k1);\n\n h1 = Long.rotateLeft(h1, 27);\n h1 += h2;\n h1 = h1 * 5 + 0x52dce729;\n\n h2 ^= mixK2(k2);\n\n h2 = Long.rotateLeft(h2, 31);\n h2 += h1;\n h2 = h2 * 5 + 0x38495ab5;\n }\n\n if (r > 0) {\n k1 = 0;\n k2 = 0;\n switch (r) {\n case 15:\n k2 ^= toLong(array[o + 14]) << 48; // fall through\n case 14:\n k2 ^= toLong(array[o + 13]) << 40; // fall through\n case 13:\n k2 ^= toLong(array[o + 12]) << 32; // fall through\n case 12:\n k2 ^= toLong(array[o + 11]) << 24; // fall through\n case 11:\n k2 ^= toLong(array[o + 10]) << 16; // fall through\n case 10:\n k2 ^= toLong(array[o + 9]) << 8; // fall through\n case 9:\n k2 ^= toLong(array[o + 8]); // fall through\n case 8:\n k1 ^= getLong(array, o);\n break;\n case 7:\n k1 ^= toLong(array[o + 6]) << 48; // fall through\n case 6:\n k1 ^= toLong(array[o + 5]) << 40; // fall through\n case 5:\n k1 ^= toLong(array[o + 4]) << 32; // fall through\n case 4:\n k1 ^= toLong(array[o + 3]) << 24; // fall through\n case 3:\n k1 ^= toLong(array[o + 2]) << 16; // fall through\n case 2:\n k1 ^= toLong(array[o + 1]) << 8; // fall through\n case 1:\n k1 ^= toLong(array[o]);\n break;\n default:\n throw new AssertionError(\"Should never get here.\");\n }\n\n h1 ^= mixK1(k1);\n h2 ^= mixK2(k2);\n }\n\n // makeHash()\n\n h1 ^= array.length;\n h2 ^= array.length;\n\n h1 += h2;\n h2 += h1;\n\n h1 = fmix64(h1);\n h2 = fmix64(h2);\n\n h1 += h2;\n //h2 += h1;\n\n // padToLong()\n return h1;\n }\n\n private static long getLong(byte[] array, int o) {\n long l = toLong(array[o + 7]) << 56;\n l |= toLong(array[o + 6]) << 48;\n l |= toLong(array[o + 5]) << 40;\n l |= toLong(array[o + 4]) << 32;\n l |= toLong(array[o + 3]) << 24;\n l |= toLong(array[o + 2]) << 16;\n l |= toLong(array[o + 1]) << 8;\n l |= toLong(array[o]);\n return l;\n }\n\n long hash(long adr, long offset, int length) {\n long o = offset;\n long r = length;\n\n long h1 = 0L;\n long h2 = 0L;\n long k1, k2;\n\n for (; r >= 16; r -= 16) {\n k1 = getLong(adr, o);\n o += 8;\n k2 = getLong(adr, o);\n o += 8;\n\n // bmix64()\n\n h1 ^= mixK1(k1);\n\n h1 = Long.rotateLeft(h1, 27);\n h1 += h2;\n h1 = h1 * 5 + 0x52dce729;\n\n h2 ^= mixK2(k2);\n\n h2 = Long.rotateLeft(h2, 31);\n h2 += h1;\n h2 = h2 * 5 + 0x38495ab5;\n }\n\n if (r > 0) {\n k1 = 0;\n k2 = 0;\n switch ((int) r) {\n case 15:\n k2 ^= toLong(Uns.getByte(adr, o + 14)) << 48; // fall through\n case 14:\n k2 ^= toLong(Uns.getByte(adr, o + 13)) << 40; // fall through\n case 13:\n k2 ^= toLong(Uns.getByte(adr, o + 12)) << 32; // fall through\n case 12:\n k2 ^= toLong(Uns.getByte(adr, o + 11)) << 24; // fall through\n case 11:\n k2 ^= toLong(Uns.getByte(adr, o + 10)) << 16; // fall through\n case 10:\n k2 ^= toLong(Uns.getByte(adr, o + 9)) << 8; // fall through\n case 9:\n k2 ^= toLong(Uns.getByte(adr, o + 8)); // fall through\n case 8:\n k1 ^= getLong(adr, o);\n break;\n case 7:\n k1 ^= toLong(Uns.getByte(adr, o + 6)) << 48; // fall through\n case 6:\n k1 ^= toLong(Uns.getByte(adr, o + 5)) << 40; // fall through\n case 5:\n k1 ^= toLong(Uns.getByte(adr, o + 4)) << 32; // fall through\n case 4:\n k1 ^= toLong(Uns.getByte(adr, o + 3)) << 24; // fall through\n case 3:\n k1 ^= toLong(Uns.getByte(adr, o + 2)) << 16; // fall through\n case 2:\n k1 ^= toLong(Uns.getByte(adr, o + 1)) << 8; // fall through\n case 1:\n k1 ^= toLong(Uns.getByte(adr, o));\n break;\n default:\n throw new AssertionError(\"Should never get here.\");\n }\n\n h1 ^= mixK1(k1);\n h2 ^= mixK2(k2);\n }\n\n // makeHash()\n\n h1 ^= length;\n h2 ^= length;\n\n h1 += h2;\n h2 += h1;\n\n h1 = fmix64(h1);\n h2 = fmix64(h2);\n\n h1 += h2;\n //h2 += h1;\n\n // padToLong()\n\n return h1;\n }\n\n private static long getLong(long adr, long o) {\n long l = toLong(Uns.getByte(adr, o + 7)) << 56;\n l |= toLong(Uns.getByte(adr, o + 6)) << 48;\n l |= toLong(Uns.getByte(adr, o + 5)) << 40;\n l |= toLong(Uns.getByte(adr, o + 4)) << 32;\n l |= toLong(Uns.getByte(adr, o + 3)) << 24;\n l |= toLong(Uns.getByte(adr, o + 2)) << 16;\n l |= toLong(Uns.getByte(adr, o + 1)) << 8;\n l |= toLong(Uns.getByte(adr, o));\n return l;\n }\n\n static final long C1 = 0x87c37b91114253d5L;\n static final long C2 = 0x4cf5ad432745937fL;\n\n static long fmix64(long k) {\n k ^= k >>> 33;\n k *= 0xff51afd7ed558ccdL;\n k ^= k >>> 33;\n k *= 0xc4ceb9fe1a85ec53L;\n k ^= k >>> 33;\n return k;\n }\n\n static long mixK1(long k1) {\n k1 *= C1;\n k1 = Long.rotateLeft(k1, 31);\n k1 *= C2;\n return k1;\n }\n\n static long mixK2(long k2) {\n k2 *= C2;\n k2 = Long.rotateLeft(k2, 33);\n k2 *= C1;\n return k2;\n }\n\n static long toLong(byte value) {\n return value & 0xff;\n }\n }\n\n static final class XxHash extends Hasher {\n\n private static final XXHashFactory xx = XXHashFactory.fastestInstance();\n\n long hash(long address, long offset, int length) {\n return xx.hash64().hash(Uns.directBufferFor(address, offset, length, true), 0);\n }\n\n long hash(byte[] array) {\n return xx.hash64().hash(array, 0, array.length, 0);\n }\n }\n}\n" }, { "path": "src/main/java/com/oath/halodb/InMemoryIndex.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport com.google.common.primitives.Ints;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport java.io.IOException;\n\n/**\n * Hash table stored in native memory, outside Java heap.\n */\nclass InMemoryIndex {\n private static final Logger logger = LoggerFactory.getLogger(InMemoryIndex.class);\n\n private final OffHeapHashTable offHeapHashTable;\n\n private final int noOfSegments;\n private final int maxSizeOfEachSegment;\n\n InMemoryIndex(int numberOfKeys, boolean useMemoryPool, int fixedKeySize, int memoryPoolChunkSize) {\n noOfSegments = Ints.checkedCast(Utils.roundUpToPowerOf2(Runtime.getRuntime().availableProcessors() * 2));\n maxSizeOfEachSegment = Ints.checkedCast(Utils.roundUpToPowerOf2(numberOfKeys / noOfSegments));\n long start = System.currentTimeMillis();\n OffHeapHashTableBuilder builder =\n OffHeapHashTableBuilder.newBuilder()\n .valueSerializer(new InMemoryIndexMetaDataSerializer())\n .segmentCount(noOfSegments)\n .hashTableSize(maxSizeOfEachSegment)\n .fixedValueSize(InMemoryIndexMetaData.SERIALIZED_SIZE)\n .loadFactor(1);\n\n if (useMemoryPool) {\n builder.useMemoryPool(true).fixedKeySize(fixedKeySize).memoryPoolChunkSize(memoryPoolChunkSize);\n }\n\n this.offHeapHashTable = builder.build();\n\n logger.debug(\"Allocated memory for the index in {}\", (System.currentTimeMillis() - start));\n }\n\n boolean put(byte[] key, InMemoryIndexMetaData metaData) {\n return offHeapHashTable.put(key, metaData);\n }\n\n boolean putIfAbsent(byte[] key, InMemoryIndexMetaData metaData) {\n return offHeapHashTable.putIfAbsent(key, metaData);\n }\n\n boolean remove(byte[] key) {\n return offHeapHashTable.remove(key);\n }\n\n boolean replace(byte[] key, InMemoryIndexMetaData oldValue, InMemoryIndexMetaData newValue) {\n return offHeapHashTable.addOrReplace(key, oldValue, newValue);\n }\n\n InMemoryIndexMetaData get(byte[] key) {\n return offHeapHashTable.get(key);\n }\n\n boolean containsKey(byte[] key) {\n return offHeapHashTable.containsKey(key);\n }\n\n void close() {\n try {\n offHeapHashTable.close();\n } catch (IOException e) {\n e.printStackTrace();\n }\n }\n\n long size() {\n return offHeapHashTable.size();\n }\n\n public OffHeapHashTableStats stats() {\n return offHeapHashTable.stats();\n }\n\n void resetStats() {\n offHeapHashTable.resetStatistics();\n }\n\n int getNoOfSegments() {\n return noOfSegments;\n }\n\n int getMaxSizeOfEachSegment() {\n return maxSizeOfEachSegment;\n }\n}\n" }, { "path": "src/main/java/com/oath/halodb/InMemoryIndexMetaData.java", "content": "\n/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport java.nio.ByteBuffer;\n\n/**\n * Metadata stored in the in-memory index for each key.\n */\nclass InMemoryIndexMetaData {\n\n private final int fileId;\n private final int valueOffset;\n private final int valueSize;\n private final long sequenceNumber;\n\n static final int SERIALIZED_SIZE = 4 + 4 + 4 + 8;\n\n InMemoryIndexMetaData(int fileId, int valueOffset, int valueSize, long sequenceNumber) {\n this.fileId = fileId;\n this.valueOffset = valueOffset;\n this.valueSize = valueSize;\n this.sequenceNumber = sequenceNumber;\n }\n\n void serialize(ByteBuffer byteBuffer) {\n byteBuffer.putInt(getFileId());\n byteBuffer.putInt(getValueOffset());\n byteBuffer.putInt(getValueSize());\n byteBuffer.putLong(getSequenceNumber());\n byteBuffer.flip();\n }\n\n static InMemoryIndexMetaData deserialize(ByteBuffer byteBuffer) {\n int fileId = byteBuffer.getInt();\n int offset = byteBuffer.getInt();\n int size = byteBuffer.getInt();\n long sequenceNumber = byteBuffer.getLong();\n\n return new InMemoryIndexMetaData(fileId, offset, size, sequenceNumber);\n }\n\n int getFileId() {\n return fileId;\n }\n\n int getValueOffset() {\n return valueOffset;\n }\n\n int getValueSize() {\n return valueSize;\n }\n\n long getSequenceNumber() {\n return sequenceNumber;\n }\n}\n" }, { "path": "src/main/java/com/oath/halodb/InMemoryIndexMetaDataSerializer.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport java.nio.ByteBuffer;\n\nclass InMemoryIndexMetaDataSerializer implements HashTableValueSerializer {\n\n public void serialize(InMemoryIndexMetaData recordMetaData, ByteBuffer byteBuffer) {\n recordMetaData.serialize(byteBuffer);\n byteBuffer.flip();\n }\n\n public InMemoryIndexMetaData deserialize(ByteBuffer byteBuffer) {\n return InMemoryIndexMetaData.deserialize(byteBuffer);\n }\n\n public int serializedSize(InMemoryIndexMetaData recordMetaData) {\n return InMemoryIndexMetaData.SERIALIZED_SIZE;\n }\n}\n" }, { "path": "src/main/java/com/oath/halodb/IndexFile.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport java.io.File;\nimport java.io.IOException;\nimport java.io.RandomAccessFile;\nimport java.nio.ByteBuffer;\nimport java.nio.channels.FileChannel;\nimport java.nio.file.Path;\nimport java.nio.file.Paths;\nimport java.util.Iterator;\nimport java.util.Objects;\n\nclass IndexFile {\n private static final Logger logger = LoggerFactory.getLogger(IndexFile.class);\n\n private final int fileId;\n private final DBDirectory dbDirectory;\n private File backingFile;\n\n private FileChannel channel;\n\n private final HaloDBOptions options;\n\n private long unFlushedData = 0;\n\n static final String INDEX_FILE_NAME = \".index\";\n private static final String nullMessage = \"Index file entry cannot be null\";\n\n IndexFile(int fileId, DBDirectory dbDirectory, HaloDBOptions options) {\n this.fileId = fileId;\n this.dbDirectory = dbDirectory;\n this.options = options;\n }\n\n void create() throws IOException {\n backingFile = getIndexFile();\n if (!backingFile.createNewFile()) {\n throw new IOException(\"Index file with id \" + fileId + \" already exists\");\n }\n channel = new RandomAccessFile(backingFile, \"rw\").getChannel();\n }\n\n void createRepairFile() throws IOException {\n backingFile = getRepairFile();\n while (!backingFile.createNewFile()) {\n logger.info(\"Repair file {} already exists, probably from a previous repair which failed. Deleting a trying again\", backingFile.getName());\n backingFile.delete();\n }\n channel = new RandomAccessFile(backingFile, \"rw\").getChannel();\n }\n\n void open() throws IOException {\n backingFile = getIndexFile();\n channel = new RandomAccessFile(backingFile, \"rw\").getChannel();\n }\n\n void close() throws IOException {\n if (channel != null) {\n channel.close();\n }\n }\n\n void delete() throws IOException {\n if (channel != null && channel.isOpen())\n channel.close();\n\n getIndexFile().delete();\n }\n\n void write(IndexFileEntry entry) throws IOException {\n Objects.requireNonNull(entry, nullMessage);\n\n ByteBuffer[] contents = entry.serialize();\n long toWrite = 0;\n for (ByteBuffer buffer : contents) {\n toWrite += buffer.remaining();\n }\n long written = 0;\n while (written < toWrite) {\n written += channel.write(contents);\n }\n\n unFlushedData += written;\n if (options.getFlushDataSizeBytes() != -1 && unFlushedData > options.getFlushDataSizeBytes()) {\n channel.force(false);\n unFlushedData = 0;\n }\n }\n\n void flushToDisk() throws IOException {\n if (channel != null && channel.isOpen())\n channel.force(true);\n }\n\n IndexFileIterator newIterator() throws IOException {\n return new IndexFileIterator();\n }\n\n Path getPath() {\n return backingFile.toPath();\n }\n\n private File getIndexFile() {\n return dbDirectory.getPath().resolve(fileId + INDEX_FILE_NAME).toFile();\n }\n\n private File getRepairFile() {\n return dbDirectory.getPath().resolve(fileId + INDEX_FILE_NAME + \".repair\").toFile();\n }\n\n public class IndexFileIterator implements Iterator {\n\n private final ByteBuffer buffer;\n\n //TODO: index files are not that large, need to check the\n // performance since we are memory mapping it.\n public IndexFileIterator() throws IOException {\n buffer = channel.map(FileChannel.MapMode.READ_ONLY, 0, channel.size());\n }\n\n @Override\n public boolean hasNext() {\n return buffer.hasRemaining();\n }\n\n @Override\n public IndexFileEntry next() {\n if (hasNext()) {\n return IndexFileEntry.deserialize(buffer);\n }\n return null;\n }\n }\n}\n" }, { "path": "src/main/java/com/oath/halodb/IndexFileEntry.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport java.nio.ByteBuffer;\nimport java.util.zip.CRC32;\n\n/**\n * This is what is stored in the index file.\n */\nclass IndexFileEntry {\n\n /**\n * checksum - 4 bytes. \n * version - 1 byte.\n * Key size - 1 bytes.\n * record size - 4 bytes.\n * record offset - 4 bytes.\n * sequence number - 8 bytes\n */\n final static int INDEX_FILE_HEADER_SIZE = 22;\n final static int CHECKSUM_SIZE = 4;\n\n static final int CHECKSUM_OFFSET = 0;\n static final int VERSION_OFFSET = 4;\n static final int KEY_SIZE_OFFSET = 5;\n static final int RECORD_SIZE_OFFSET = 6;\n static final int RECORD_OFFSET = 10;\n static final int SEQUENCE_NUMBER_OFFSET = 14;\n\n\n private final byte[] key;\n private final int recordSize;\n private final int recordOffset;\n private final byte keySize;\n private final int version;\n private final long sequenceNumber;\n private final long checkSum;\n\n IndexFileEntry(byte[] key, int recordSize, int recordOffset, long sequenceNumber, int version, long checkSum) {\n this.key = key;\n this.recordSize = recordSize;\n this.recordOffset = recordOffset;\n this.sequenceNumber = sequenceNumber;\n this.version = version;\n this.checkSum = checkSum;\n\n this.keySize = (byte)key.length;\n }\n\n ByteBuffer[] serialize() {\n byte[] header = new byte[INDEX_FILE_HEADER_SIZE];\n ByteBuffer h = ByteBuffer.wrap(header);\n\n h.put(VERSION_OFFSET, (byte)version);\n h.put(KEY_SIZE_OFFSET, keySize);\n h.putInt(RECORD_SIZE_OFFSET, recordSize);\n h.putInt(RECORD_OFFSET, recordOffset);\n h.putLong(SEQUENCE_NUMBER_OFFSET, sequenceNumber);\n long crc32 = computeCheckSum(h.array());\n h.putInt(CHECKSUM_OFFSET, Utils.toSignedIntFromLong(crc32));\n\n return new ByteBuffer[] { h, ByteBuffer.wrap(key) };\n }\n\n static IndexFileEntry deserialize(ByteBuffer buffer) {\n long crc32 = Utils.toUnsignedIntFromInt(buffer.getInt());\n int version = Utils.toUnsignedByte(buffer.get());\n byte keySize = buffer.get();\n int recordSize = buffer.getInt();\n int offset = buffer.getInt();\n long sequenceNumber = buffer.getLong();\n\n byte[] key = new byte[keySize];\n buffer.get(key);\n\n return new IndexFileEntry(key, recordSize, offset, sequenceNumber, version, crc32);\n }\n\n static IndexFileEntry deserializeIfNotCorrupted(ByteBuffer buffer) {\n if (buffer.remaining() < INDEX_FILE_HEADER_SIZE) {\n return null;\n }\n\n long crc32 = Utils.toUnsignedIntFromInt(buffer.getInt());\n int version = Utils.toUnsignedByte(buffer.get());\n byte keySize = buffer.get();\n int recordSize = buffer.getInt();\n int offset = buffer.getInt();\n long sequenceNumber = buffer.getLong();\n if (sequenceNumber < 0 || keySize <= 0\n || version < 0 || version > 255\n || recordSize <= 0 || offset < 0\n || buffer.remaining() < keySize) {\n return null;\n }\n\n byte[] key = new byte[keySize];\n buffer.get(key);\n\n IndexFileEntry entry = new IndexFileEntry(key, recordSize, offset, sequenceNumber, version, crc32);\n if (entry.computeCheckSum() != entry.checkSum) {\n return null;\n }\n\n return entry;\n }\n\n private long computeCheckSum(byte[] header) {\n CRC32 crc32 = new CRC32();\n crc32.update(header, CHECKSUM_OFFSET + CHECKSUM_SIZE, INDEX_FILE_HEADER_SIZE - CHECKSUM_SIZE);\n crc32.update(key);\n return crc32.getValue();\n }\n\n long computeCheckSum() {\n ByteBuffer header = ByteBuffer.allocate(INDEX_FILE_HEADER_SIZE);\n header.put(VERSION_OFFSET, (byte)version);\n header.put(KEY_SIZE_OFFSET, keySize);\n header.putInt(RECORD_SIZE_OFFSET, recordSize);\n header.putInt(RECORD_OFFSET, recordOffset);\n header.putLong(SEQUENCE_NUMBER_OFFSET, sequenceNumber);\n return computeCheckSum(header.array());\n }\n\n byte[] getKey() {\n return key;\n }\n\n int getRecordSize() {\n return recordSize;\n }\n\n int getRecordOffset() {\n return recordOffset;\n }\n\n long getSequenceNumber() {\n return sequenceNumber;\n }\n\n int getVersion() {\n return version;\n }\n\n long getCheckSum() {\n return checkSum;\n }\n}\n" }, { "path": "src/main/java/com/oath/halodb/JNANativeAllocator.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\n// This code is a derivative work heavily modified from the OHC project. See NOTICE file for copyright and license.\n\npackage com.oath.halodb;\n\nimport com.sun.jna.Native;\n\nfinal class JNANativeAllocator implements NativeMemoryAllocator {\n\n public long allocate(long size) {\n try {\n return Native.malloc(size);\n } catch (OutOfMemoryError oom) {\n return 0L;\n }\n }\n\n public void free(long peer) {\n Native.free(peer);\n }\n\n public long getTotalAllocated() {\n return -1L;\n }\n}\n" }, { "path": "src/main/java/com/oath/halodb/KeyBuffer.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\n// This code is a derivative work heavily modified from the OHC project. See NOTICE file for copyright and license.\n\npackage com.oath.halodb;\n\nimport java.util.Arrays;\n\nfinal class KeyBuffer {\n\n final byte[] buffer;\n private long hash;\n\n KeyBuffer(byte[] buffer) {\n this.buffer = buffer;\n }\n\n long hash() {\n return hash;\n }\n\n KeyBuffer finish(Hasher hasher) {\n hash = hasher.hash(buffer);\n\n return this;\n }\n\n public boolean equals(Object o) {\n if (this == o) {\n return true;\n }\n if (o == null || getClass() != o.getClass()) {\n return false;\n }\n\n KeyBuffer keyBuffer = (KeyBuffer) o;\n\n return buffer.length == keyBuffer.buffer.length && Arrays.equals(keyBuffer.buffer, buffer);\n }\n\n public int size() {\n return buffer.length;\n }\n\n public int hashCode() {\n return (int) hash;\n }\n\n private static String pad(int val) {\n String str = Integer.toHexString(val & 0xff);\n while (str.length() == 1) {\n str = '0' + str;\n }\n return str;\n }\n\n @Override\n public String toString() {\n byte[] b = buffer;\n StringBuilder sb = new StringBuilder(b.length * 3);\n for (int ii = 0; ii < b.length; ii++) {\n if (ii % 8 == 0 && ii != 0) {\n sb.append('\\n');\n }\n sb.append(pad(b[ii]));\n sb.append(' ');\n }\n return sb.toString();\n }\n\n // This is meant to be used only with non-pooled memory.\n //TODO: move to another class. \n boolean sameKey(long hashEntryAdr) {\n long serKeyLen = NonMemoryPoolHashEntries.getKeyLen(hashEntryAdr);\n return serKeyLen == buffer.length && compareKey(hashEntryAdr);\n }\n\n private boolean compareKey(long hashEntryAdr) {\n int blkOff = (int) NonMemoryPoolHashEntries.ENTRY_OFF_DATA;\n int p = 0;\n int endIdx = buffer.length;\n for (; endIdx - p >= 8; p += 8) {\n if (Uns.getLong(hashEntryAdr, blkOff + p) != Uns.getLongFromByteArray(buffer, p)) {\n return false;\n }\n }\n for (; endIdx - p >= 4; p += 4) {\n if (Uns.getInt(hashEntryAdr, blkOff + p) != Uns.getIntFromByteArray(buffer, p)) {\n return false;\n }\n }\n for (; endIdx - p >= 2; p += 2) {\n if (Uns.getShort(hashEntryAdr, blkOff + p) != Uns.getShortFromByteArray(buffer, p)) {\n return false;\n }\n }\n for (; endIdx - p >= 1; p += 1) {\n if (Uns.getByte(hashEntryAdr, blkOff + p) != buffer[p]) {\n return false;\n }\n }\n\n return true;\n }\n}\n" }, { "path": "src/main/java/com/oath/halodb/LongArrayList.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\n// This code is a derivative work heavily modified from the OHC project. See NOTICE file for copyright and license.\n\npackage com.oath.halodb;\n\nimport java.util.Arrays;\n\nfinal class LongArrayList {\n\n private long[] array;\n private int size;\n\n public LongArrayList() {\n this(10);\n }\n\n public LongArrayList(int initialCapacity) {\n array = new long[initialCapacity];\n }\n\n public long getLong(int i) {\n if (i < 0 || i >= size) {\n throw new ArrayIndexOutOfBoundsException();\n }\n return array[i];\n }\n\n public void clear() {\n size = 0;\n }\n\n public int size() {\n return size;\n }\n\n public void add(long value) {\n if (size == array.length) {\n array = Arrays.copyOf(array, size * 2);\n }\n array[size++] = value;\n }\n}\n" }, { "path": "src/main/java/com/oath/halodb/MemoryPoolAddress.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\n/**\n * Represents the address of an entry in the memory pool. It will have two components: the index of the chunk which\n * contains the entry and the offset within the chunk.\n */\nclass MemoryPoolAddress {\n\n final byte chunkIndex;\n final int chunkOffset;\n\n MemoryPoolAddress(byte chunkIndex, int chunkOffset) {\n this.chunkIndex = chunkIndex;\n this.chunkOffset = chunkOffset;\n }\n\n @Override\n public boolean equals(Object o) {\n if (o == this) {\n return true;\n }\n if (!(o instanceof MemoryPoolAddress)) {\n return false;\n }\n MemoryPoolAddress m = (MemoryPoolAddress) o;\n return m.chunkIndex == chunkIndex && m.chunkOffset == chunkOffset;\n }\n\n @Override\n public int hashCode() {\n return 31 * ((31 * chunkIndex) + chunkOffset);\n }\n}\n" }, { "path": "src/main/java/com/oath/halodb/MemoryPoolChunk.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport java.nio.ByteBuffer;\n\nimport static com.oath.halodb.MemoryPoolHashEntries.*;\n\n/**\n * Memory pool is divided into chunks of configurable size. This represents such a chunk.\n */\nclass MemoryPoolChunk {\n\n private final long address;\n private final int chunkSize;\n private final int fixedKeyLength;\n private final int fixedValueLength;\n private final int fixedSlotSize;\n private int writeOffset = 0;\n\n private MemoryPoolChunk(long address, int chunkSize, int fixedKeyLength, int fixedValueLength) {\n this.address = address;\n this.chunkSize = chunkSize;\n this.fixedKeyLength = fixedKeyLength;\n this.fixedValueLength = fixedValueLength;\n this.fixedSlotSize = HEADER_SIZE + fixedKeyLength + fixedValueLength;\n }\n\n static MemoryPoolChunk create(int chunkSize, int fixedKeyLength, int fixedValueLength) {\n int fixedSlotSize = HEADER_SIZE + fixedKeyLength + fixedValueLength;\n if (fixedSlotSize > chunkSize) {\n throw new IllegalArgumentException(\"fixedSlotSize \" + fixedSlotSize + \" must be smaller than chunkSize \" + chunkSize);\n }\n long address = Uns.allocate(chunkSize, true);\n return new MemoryPoolChunk(address, chunkSize, fixedKeyLength, fixedValueLength);\n }\n\n void destroy() {\n Uns.free(address);\n }\n\n MemoryPoolAddress getNextAddress(int slotOffset) {\n byte chunkIndex = Uns.getByte(address, slotOffset + ENTRY_OFF_NEXT_CHUNK_INDEX);\n int chunkOffset = Uns.getInt(address, slotOffset + ENTRY_OFF_NEXT_CHUNK_OFFSET);\n\n return new MemoryPoolAddress(chunkIndex, chunkOffset);\n }\n\n void setNextAddress(int slotOffset, MemoryPoolAddress next) {\n Uns.putByte(address, slotOffset + ENTRY_OFF_NEXT_CHUNK_INDEX, next.chunkIndex);\n Uns.putInt(address, slotOffset + ENTRY_OFF_NEXT_CHUNK_OFFSET, next.chunkOffset);\n }\n\n /**\n * Relative put method. Writes to the slot pointed to by the writeOffset and increments the writeOffset.\n */\n void fillNextSlot(byte[] key, byte[] value, MemoryPoolAddress nextAddress) {\n fillSlot(writeOffset, key, value, nextAddress);\n writeOffset += fixedSlotSize;\n }\n\n /**\n * Absolute put method. Writes to the slot pointed to by the offset.\n */\n void fillSlot(int slotOffset, byte[] key, byte[] value, MemoryPoolAddress nextAddress) {\n if (key.length > fixedKeyLength || value.length != fixedValueLength) {\n throw new IllegalArgumentException(\n String.format(\"Invalid request. Key length %d. fixed key length %d. Value length %d\",\n key.length, fixedKeyLength, value.length)\n );\n }\n if (chunkSize - slotOffset < fixedSlotSize) {\n throw new IllegalArgumentException(\n String.format(\"Invalid offset %d. Chunk size %d. fixed slot size %d\",\n slotOffset, chunkSize, fixedSlotSize)\n );\n }\n\n setNextAddress(slotOffset, nextAddress);\n Uns.putByte(address, slotOffset + ENTRY_OFF_KEY_LENGTH, (byte) key.length);\n Uns.copyMemory(key, 0, address, slotOffset + ENTRY_OFF_DATA, key.length);\n setValue(value, slotOffset);\n }\n\n void setValue(byte[] value, int slotOffset) {\n if (value.length != fixedValueLength) {\n throw new IllegalArgumentException(\n String.format(\"Invalid value length. fixedValueLength %d, value length %d\",\n fixedValueLength, value.length)\n );\n }\n\n Uns.copyMemory(value, 0, address, slotOffset + ENTRY_OFF_DATA + fixedKeyLength, value.length);\n }\n\n int getWriteOffset() {\n return writeOffset;\n }\n\n int remaining() {\n return chunkSize - writeOffset;\n }\n\n ByteBuffer readOnlyValueByteBuffer(int offset) {\n return Uns.directBufferFor(address, offset + ENTRY_OFF_DATA + fixedKeyLength, fixedValueLength, true);\n }\n\n ByteBuffer readOnlyKeyByteBuffer(int offset) {\n return Uns.directBufferFor(address, offset + ENTRY_OFF_DATA, getKeyLength(offset), true);\n }\n\n long computeHash(int slotOffset, Hasher hasher) {\n return hasher.hash(address, slotOffset + ENTRY_OFF_DATA, getKeyLength(slotOffset));\n }\n\n\n boolean compareKey(int slotOffset, byte[] key) {\n if (key.length > fixedKeyLength || slotOffset + fixedSlotSize > chunkSize) {\n throw new IllegalArgumentException(\"Invalid request. slotOffset - \" + slotOffset + \" key.length - \" + key.length);\n }\n\n return getKeyLength(slotOffset) == key.length && compare(slotOffset + ENTRY_OFF_DATA, key);\n }\n\n boolean compareValue(int slotOffset, byte[] value) {\n if (value.length != fixedValueLength || slotOffset + fixedSlotSize > chunkSize) {\n throw new IllegalArgumentException(\"Invalid request. slotOffset - \" + slotOffset + \" value.length - \" + value.length);\n }\n\n return compare(slotOffset + ENTRY_OFF_DATA + fixedKeyLength, value);\n }\n\n private boolean compare(int offset, byte[] array) {\n int p = 0, length = array.length;\n for (; length - p >= 8; p += 8) {\n if (Uns.getLong(address, offset + p) != Uns.getLongFromByteArray(array, p)) {\n return false;\n }\n }\n for (; length - p >= 4; p += 4) {\n if (Uns.getInt(address, offset + p) != Uns.getIntFromByteArray(array, p)) {\n return false;\n }\n }\n for (; length - p >= 2; p += 2) {\n if (Uns.getShort(address, offset + p) != Uns.getShortFromByteArray(array, p)) {\n return false;\n }\n }\n for (; length - p >= 1; p += 1) {\n if (Uns.getByte(address, offset + p) != array[p]) {\n return false;\n }\n }\n\n return true;\n }\n\n private byte getKeyLength(int slotOffset) {\n return Uns.getByte(address, slotOffset + ENTRY_OFF_KEY_LENGTH);\n }\n}\n" }, { "path": "src/main/java/com/oath/halodb/MemoryPoolHashEntries.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nclass MemoryPoolHashEntries {\n\n /*\n * chunk index - 1 byte.\n * chunk offset - 4 byte.\n * key length - 1 byte.\n */\n static final int HEADER_SIZE = 1 + 4 + 1;\n\n static final int ENTRY_OFF_NEXT_CHUNK_INDEX = 0;\n static final int ENTRY_OFF_NEXT_CHUNK_OFFSET = 1;\n\n // offset of key length (1 bytes, byte)\n static final int ENTRY_OFF_KEY_LENGTH = 5;\n\n // offset of data in first block\n static final int ENTRY_OFF_DATA = 6;\n\n}\n" }, { "path": "src/main/java/com/oath/halodb/NativeMemoryAllocator.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\n//This code is a derivative work heavily modified from the OHC project. See NOTICE file for copyright and license.\n\npackage com.oath.halodb;\n\ninterface NativeMemoryAllocator {\n\n long allocate(long size);\n void free(long peer);\n long getTotalAllocated();\n}\n" }, { "path": "src/main/java/com/oath/halodb/NonMemoryPoolHashEntries.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\n// This code is a derivative work heavily modified from the OHC project. See NOTICE file for copyright and license.\n\npackage com.oath.halodb;\n\n/**\n * Encapsulates access to hash entries.\n */\nfinal class NonMemoryPoolHashEntries {\n\n // offset of next hash entry in a hash bucket (8 bytes, long)\n static final long ENTRY_OFF_NEXT = 0;\n\n // offset of key length (1 bytes, byte)\n static final long ENTRY_OFF_KEY_LENGTH = 8;\n\n // offset of data in first block\n static final long ENTRY_OFF_DATA = 9;\n\n static void init(int keyLen, long hashEntryAdr) {\n setNext(hashEntryAdr, 0L);\n Uns.putByte(hashEntryAdr, ENTRY_OFF_KEY_LENGTH, (byte) keyLen);\n }\n\n static long getNext(long hashEntryAdr) {\n return hashEntryAdr != 0L ? Uns.getLong(hashEntryAdr, ENTRY_OFF_NEXT) : 0L;\n }\n\n static void setNext(long hashEntryAdr, long nextAdr) {\n if (hashEntryAdr == nextAdr) {\n throw new IllegalArgumentException();\n }\n if (hashEntryAdr != 0L) {\n Uns.putLong(hashEntryAdr, ENTRY_OFF_NEXT, nextAdr);\n }\n }\n\n static int getKeyLen(long hashEntryAdr) {\n return Uns.getByte(hashEntryAdr, ENTRY_OFF_KEY_LENGTH);\n }\n}\n" }, { "path": "src/main/java/com/oath/halodb/OffHeapHashTable.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\n// This code is a derivative work heavily modified from the OHC project. See NOTICE file for copyright and license.\n\npackage com.oath.halodb;\n\nimport com.oath.halodb.histo.EstimatedHistogram;\n\nimport java.io.Closeable;\n\ninterface OffHeapHashTable extends Closeable {\n\n /**\n * @param key key of the entry to be added. Must not be {@code null}.\n * @param value value of the entry to be added. Must not be {@code null}.\n * @return {@code true}, if the entry has been added, {@code false} otherwise\n */\n boolean put(byte[] key, V value);\n\n /**\n * Adds key/value if either the key is not present and {@code old} is null or the existing value matches parameter {@code old}.\n *\n * @param key key of the entry to be added or replaced. Must not be {@code null}.\n * @param old if the entry exists, it's serialized value is compared to the serialized value of {@code old}\n * and only replaced, if it matches.\n * @param value value of the entry to be added. Must not be {@code null}.\n * @return {@code true} on success or {@code false} if the existing value does not matcuh {@code old}\n */\n boolean addOrReplace(byte[] key, V old, V value);\n\n /**\n * @param key key of the entry to be added. Must not be {@code null}.\n * @param value value of the entry to be added. Must not be {@code null}.\n * @return {@code true} on success or {@code false} if the key is already present.\n */\n boolean putIfAbsent(byte[] key, V value);\n\n /**\n * Remove a single entry for the given key.\n *\n * @param key key of the entry to be removed. Must not be {@code null}.\n * @return {@code true}, if the entry has been removed, {@code false} otherwise\n */\n boolean remove(byte[] key);\n\n /**\n * Removes all entries from the cache.\n */\n void clear();\n\n /**\n * Get the value for a given key.\n *\n * @param key key of the entry to be retrieved. Must not be {@code null}.\n * @return either the non-{@code null} value or {@code null} if no entry for the requested key exists\n */\n V get(byte[] key);\n\n /**\n * Checks whether an entry for a given key exists.\n * Usually, this is more efficient than testing for {@code null} via {@link #get(Object)}.\n *\n * @param key key of the entry to be retrieved. Must not be {@code null}.\n * @return either {@code true} if an entry for the given key exists or {@code false} if no entry for the requested key exists\n */\n boolean containsKey(byte[] key);\n\n // statistics / information\n\n void resetStatistics();\n\n long size();\n\n int[] hashTableSizes();\n\n SegmentStats[] perSegmentStats();\n\n EstimatedHistogram getBucketHistogram();\n\n int segments();\n\n float loadFactor();\n\n OffHeapHashTableStats stats();\n}\n" }, { "path": "src/main/java/com/oath/halodb/OffHeapHashTableBuilder.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\n// This code is a derivative work heavily modified from the OHC project. See NOTICE file for copyright and license.\n\npackage com.oath.halodb;\n\nclass OffHeapHashTableBuilder {\n\n private int segmentCount;\n private int hashTableSize = 8192;\n private int memoryPoolChunkSize = 2 * 1024 * 1024;\n private HashTableValueSerializer valueSerializer;\n private float loadFactor = .75f;\n private int fixedKeySize = -1;\n private int fixedValueSize = -1;\n private HashAlgorithm hashAlgorighm = HashAlgorithm.MURMUR3;\n private Hasher hasher;\n private boolean unlocked;\n private boolean useMemoryPool = false;\n\n private OffHeapHashTableBuilder() {\n int cpus = Runtime.getRuntime().availableProcessors();\n\n segmentCount = roundUpToPowerOf2(cpus * 2, 1 << 30);\n }\n\n static final String SYSTEM_PROPERTY_PREFIX = \"org.caffinitas.ohc.\";\n\n static int roundUpToPowerOf2(int number, int max) {\n return number >= max\n ? max\n : (number > 1) ? Integer.highestOneBit((number - 1) << 1) : 1;\n }\n\n static OffHeapHashTableBuilder newBuilder() {\n return new OffHeapHashTableBuilder<>();\n }\n\n public OffHeapHashTable build() {\n if (fixedValueSize == -1) {\n throw new IllegalArgumentException(\"Need to set fixedValueSize\");\n }\n\n //TODO: write a test.\n if (useMemoryPool && fixedKeySize == -1) {\n throw new IllegalArgumentException(\"Need to set fixedKeySize when using memory pool\");\n }\n\n if (valueSerializer == null) {\n throw new IllegalArgumentException(\"Value serializer must be set.\");\n }\n\n return new OffHeapHashTableImpl<>(this);\n }\n\n public int getHashTableSize() {\n return hashTableSize;\n }\n\n public OffHeapHashTableBuilder hashTableSize(int hashTableSize) {\n if (hashTableSize < -1) {\n throw new IllegalArgumentException(\"hashTableSize:\" + hashTableSize);\n }\n this.hashTableSize = hashTableSize;\n return this;\n }\n\n public int getMemoryPoolChunkSize() {\n return memoryPoolChunkSize;\n }\n\n public OffHeapHashTableBuilder memoryPoolChunkSize(int chunkSize) {\n if (chunkSize < -1) {\n throw new IllegalArgumentException(\"memoryPoolChunkSize:\" + chunkSize);\n }\n this.memoryPoolChunkSize = chunkSize;\n return this;\n }\n\n public HashTableValueSerializer getValueSerializer() {\n return valueSerializer;\n }\n\n public OffHeapHashTableBuilder valueSerializer(HashTableValueSerializer valueSerializer) {\n this.valueSerializer = valueSerializer;\n return this;\n }\n\n public int getSegmentCount() {\n return segmentCount;\n }\n\n public OffHeapHashTableBuilder segmentCount(int segmentCount) {\n if (segmentCount < -1) {\n throw new IllegalArgumentException(\"segmentCount:\" + segmentCount);\n }\n this.segmentCount = segmentCount;\n return this;\n }\n\n public float getLoadFactor() {\n return loadFactor;\n }\n\n public OffHeapHashTableBuilder loadFactor(float loadFactor) {\n if (loadFactor <= 0f) {\n throw new IllegalArgumentException(\"loadFactor:\" + loadFactor);\n }\n this.loadFactor = loadFactor;\n return this;\n }\n\n public int getFixedKeySize() {\n return fixedKeySize;\n }\n\n public OffHeapHashTableBuilder fixedKeySize(int fixedKeySize) {\n if (fixedKeySize <= 0) {\n throw new IllegalArgumentException(\"fixedValueSize:\" + fixedKeySize);\n }\n this.fixedKeySize = fixedKeySize;\n return this;\n }\n\n public int getFixedValueSize() {\n return fixedValueSize;\n }\n\n public OffHeapHashTableBuilder fixedValueSize(int fixedValueSize) {\n if (fixedValueSize <= 0) {\n throw new IllegalArgumentException(\"fixedValueSize:\" + fixedValueSize);\n }\n this.fixedValueSize = fixedValueSize;\n return this;\n }\n\n public HashAlgorithm getHashAlgorighm() {\n return hashAlgorighm;\n }\n\n public Hasher getHasher() {\n return hasher;\n }\n\n public OffHeapHashTableBuilder hashMode(HashAlgorithm hashMode) {\n if (hashMode == null) {\n throw new NullPointerException(\"hashMode\");\n }\n this.hashAlgorighm = hashMode;\n this.hasher = Hasher.create(hashMode);\n return this;\n }\n\n public boolean isUnlocked() {\n return unlocked;\n }\n\n public OffHeapHashTableBuilder unlocked(boolean unlocked) {\n this.unlocked = unlocked;\n return this;\n }\n\n public boolean isUseMemoryPool() {\n return useMemoryPool;\n }\n\n public OffHeapHashTableBuilder useMemoryPool(boolean useMemoryPool) {\n this.useMemoryPool = useMemoryPool;\n return this;\n }\n}\n" }, { "path": "src/main/java/com/oath/halodb/OffHeapHashTableImpl.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\n// This code is a derivative work heavily modified from the OHC project. See NOTICE file for copyright and license.\n\npackage com.oath.halodb;\n\nimport com.google.common.primitives.Ints;\nimport com.oath.halodb.histo.EstimatedHistogram;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport java.util.ArrayList;\nimport java.util.Arrays;\nimport java.util.Collections;\nimport java.util.List;\n\nfinal class OffHeapHashTableImpl implements OffHeapHashTable {\n\n private static final Logger logger = LoggerFactory.getLogger(OffHeapHashTableImpl.class);\n\n private final HashTableValueSerializer valueSerializer;\n\n private final int fixedValueLength;\n\n private final List> segments;\n private final long segmentMask;\n private final int segmentShift;\n\n private final int segmentCount;\n\n private volatile long putFailCount;\n\n private boolean closed;\n\n private final Hasher hasher;\n\n OffHeapHashTableImpl(OffHeapHashTableBuilder builder) {\n this.hasher = Hasher.create(builder.getHashAlgorighm());\n this.fixedValueLength = builder.getFixedValueSize();\n\n // build segments\n if (builder.getSegmentCount() <= 0) {\n throw new IllegalArgumentException(\"Segment count should be > 0\");\n }\n segmentCount = Ints.checkedCast(HashTableUtil.roundUpToPowerOf2(builder.getSegmentCount(), 1 << 30));\n segments = new ArrayList<>(segmentCount);\n for (int i = 0; i < segmentCount; i++) {\n try {\n segments.add(allocateSegment(builder));\n } catch (RuntimeException e) {\n for (; i >= 0; i--) {\n if (segments.get(i) != null) {\n segments.get(i).release();\n }\n }\n throw e;\n }\n }\n\n // bit-mask for segment part of hash\n int bitNum = HashTableUtil.bitNum(segmentCount) - 1;\n this.segmentShift = 64 - bitNum;\n this.segmentMask = ((long) segmentCount - 1) << segmentShift;\n\n this.valueSerializer = builder.getValueSerializer();\n if (valueSerializer == null) {\n throw new NullPointerException(\"valueSerializer == null\");\n }\n\n logger.debug(\"off-heap index with {} segments created.\", segmentCount);\n }\n\n private Segment allocateSegment(OffHeapHashTableBuilder builder) {\n if (builder.isUseMemoryPool()) {\n return new SegmentWithMemoryPool<>(builder);\n }\n return new SegmentNonMemoryPool<>(builder);\n }\n\n public V get(byte[] key) {\n if (key == null) {\n throw new NullPointerException();\n }\n\n KeyBuffer keySource = keySource(key);\n return segment(keySource.hash()).getEntry(keySource);\n }\n\n public boolean containsKey(byte[] key) {\n if (key == null) {\n throw new NullPointerException();\n }\n\n KeyBuffer keySource = keySource(key);\n return segment(keySource.hash()).containsEntry(keySource);\n }\n\n public boolean put(byte[] k, V v) {\n return putInternal(k, v, false, null);\n }\n\n public boolean addOrReplace(byte[] key, V old, V value) {\n return putInternal(key, value, false, old);\n }\n\n public boolean putIfAbsent(byte[] k, V v) {\n return putInternal(k, v, true, null);\n }\n\n private boolean putInternal(byte[] key, V value, boolean ifAbsent, V old) {\n if (key == null || value == null) {\n throw new NullPointerException();\n }\n\n int valueSize = valueSize(value);\n if (valueSize != fixedValueLength) {\n throw new IllegalArgumentException(\"value size \" + valueSize + \" greater than fixed value size \" + fixedValueLength);\n }\n\n if (old != null && valueSize(old) != fixedValueLength) {\n throw new IllegalArgumentException(\"old value size \" + valueSize(old) + \" greater than fixed value size \" + fixedValueLength);\n }\n\n if (key.length > Byte.MAX_VALUE) {\n throw new IllegalArgumentException(\"key size of \" + key.length + \" exceeds max permitted size of \" + Byte.MAX_VALUE);\n }\n\n long hash = hasher.hash(key);\n return segment(hash).putEntry(key, value, hash, ifAbsent, old);\n }\n\n private int valueSize(V v) {\n int sz = valueSerializer.serializedSize(v);\n if (sz <= 0) {\n throw new IllegalArgumentException(\"Illegal value length \" + sz);\n }\n return sz;\n }\n\n public boolean remove(byte[] k) {\n if (k == null) {\n throw new NullPointerException();\n }\n\n KeyBuffer keySource = keySource(k);\n return segment(keySource.hash()).removeEntry(keySource);\n }\n\n private Segment segment(long hash) {\n int seg = (int) ((hash & segmentMask) >>> segmentShift);\n return segments.get(seg);\n }\n\n private KeyBuffer keySource(byte[] key) {\n KeyBuffer keyBuffer = new KeyBuffer(key);\n return keyBuffer.finish(hasher);\n }\n\n //\n // maintenance\n //\n\n public void clear() {\n for (Segment map : segments) {\n map.clear();\n }\n }\n\n //\n // state\n //\n\n //TODO: remove.\n public void setCapacity(long capacity) {\n\n }\n\n public void close() {\n closed = true;\n for (Segment map : segments) {\n map.release();\n }\n Collections.fill(segments, null);\n\n if (logger.isDebugEnabled()) {\n logger.debug(\"Closing OHC instance\");\n }\n }\n\n //\n // statistics and related stuff\n //\n\n public void resetStatistics() {\n for (Segment map : segments) {\n map.resetStatistics();\n }\n putFailCount = 0;\n }\n\n public OffHeapHashTableStats stats() {\n long hitCount = 0, missCount = 0, size = 0,\n freeCapacity = 0, rehashes = 0, putAddCount = 0, putReplaceCount = 0, removeCount = 0;\n for (Segment map : segments) {\n hitCount += map.hitCount();\n missCount += map.missCount();\n size += map.size();\n rehashes += map.rehashes();\n putAddCount += map.putAddCount();\n putReplaceCount += map.putReplaceCount();\n removeCount += map.removeCount();\n }\n\n return new OffHeapHashTableStats(\n hitCount,\n missCount,\n size,\n rehashes,\n putAddCount,\n putReplaceCount,\n putFailCount,\n removeCount,\n perSegmentStats());\n }\n\n public long size() {\n long size = 0L;\n for (Segment map : segments) {\n size += map.size();\n }\n return size;\n }\n\n public int segments() {\n return segments.size();\n }\n\n public float loadFactor() {\n return segments.get(0).loadFactor();\n }\n\n public int[] hashTableSizes() {\n int[] r = new int[segments.size()];\n for (int i = 0; i < segments.size(); i++) {\n r[i] = segments.get(i).hashTableSize();\n }\n return r;\n }\n\n public long[] perSegmentSizes() {\n long[] r = new long[segments.size()];\n for (int i = 0; i < segments.size(); i++) {\n r[i] = segments.get(i).size();\n }\n return r;\n }\n\n public SegmentStats[] perSegmentStats() {\n SegmentStats[] stats = new SegmentStats[segments.size()];\n for (int i = 0; i < stats.length; i++) {\n Segment map = segments.get(i);\n stats[i] = new SegmentStats(map.size(), map.numberOfChunks(), map.numberOfSlots(), map.freeListSize());\n }\n\n return stats;\n }\n\n public EstimatedHistogram getBucketHistogram() {\n EstimatedHistogram hist = new EstimatedHistogram();\n for (Segment map : segments) {\n map.updateBucketHistogram(hist);\n }\n\n long[] offsets = hist.getBucketOffsets();\n long[] buckets = hist.getBuckets(false);\n\n for (int i = buckets.length - 1; i > 0; i--) {\n if (buckets[i] != 0L) {\n offsets = Arrays.copyOf(offsets, i + 2);\n buckets = Arrays.copyOf(buckets, i + 3);\n System.arraycopy(offsets, 0, offsets, 1, i + 1);\n System.arraycopy(buckets, 0, buckets, 1, i + 2);\n offsets[0] = 0L;\n buckets[0] = 0L;\n break;\n }\n }\n\n for (int i = 0; i < offsets.length; i++) {\n offsets[i]--;\n }\n\n return new EstimatedHistogram(offsets, buckets);\n }\n\n public String toString() {\n return getClass().getSimpleName() + \" ,segments=\" + segments.size();\n }\n}\n" }, { "path": "src/main/java/com/oath/halodb/OffHeapHashTableStats.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\n// This code is a derivative work heavily modified from the OHC project. See NOTICE file for copyright and license.\n\npackage com.oath.halodb;\n\nimport com.google.common.base.Objects;\n\nfinal class OffHeapHashTableStats {\n\n private final long hitCount;\n private final long missCount;\n private final long size;\n private final long rehashCount;\n private final long putAddCount;\n private final long putReplaceCount;\n private final long putFailCount;\n private final long removeCount;\n private final SegmentStats[] segmentStats;\n\n public OffHeapHashTableStats(long hitCount, long missCount,\n long size, long rehashCount,\n long putAddCount, long putReplaceCount, long putFailCount, long removeCount,\n SegmentStats[] segmentStats) {\n this.hitCount = hitCount;\n this.missCount = missCount;\n this.size = size;\n this.rehashCount = rehashCount;\n this.putAddCount = putAddCount;\n this.putReplaceCount = putReplaceCount;\n this.putFailCount = putFailCount;\n this.removeCount = removeCount;\n this.segmentStats = segmentStats;\n }\n\n public long getRehashCount() {\n return rehashCount;\n }\n\n public long getHitCount() {\n return hitCount;\n }\n\n public long getMissCount() {\n return missCount;\n }\n\n public long getSize() {\n return size;\n }\n\n public long getPutAddCount() {\n return putAddCount;\n }\n\n public long getPutReplaceCount() {\n return putReplaceCount;\n }\n\n public long getPutFailCount() {\n return putFailCount;\n }\n\n public long getRemoveCount() {\n return removeCount;\n }\n\n public SegmentStats[] getSegmentStats() {\n return segmentStats;\n }\n\n public String toString() {\n return Objects.toStringHelper(this)\n .add(\"hitCount\", hitCount)\n .add(\"missCount\", missCount)\n .add(\"size\", size)\n .add(\"rehashCount\", rehashCount)\n .add(\"put(add/replace/fail)\", Long.toString(putAddCount) + '/' + putReplaceCount + '/' + putFailCount)\n .add(\"removeCount\", removeCount)\n .toString();\n }\n\n private static long maxOf(long[] arr) {\n long r = 0;\n for (long l : arr) {\n if (l > r) {\n r = l;\n }\n }\n return r;\n }\n\n private static long minOf(long[] arr) {\n long r = Long.MAX_VALUE;\n for (long l : arr) {\n if (l < r) {\n r = l;\n }\n }\n return r;\n }\n\n private static double avgOf(long[] arr) {\n double r = 0d;\n for (long l : arr) {\n r += l;\n }\n return r / arr.length;\n }\n\n public boolean equals(Object o) {\n if (this == o) return true;\n if (o == null || getClass() != o.getClass()) return false;\n\n OffHeapHashTableStats that = (OffHeapHashTableStats) o;\n\n if (hitCount != that.hitCount) return false;\n if (missCount != that.missCount) return false;\n if (putAddCount != that.putAddCount) return false;\n if (putFailCount != that.putFailCount) return false;\n if (putReplaceCount != that.putReplaceCount) return false;\n// if (rehashCount != that.rehashCount) return false;\n if (removeCount != that.removeCount) return false;\n if (size != that.size) return false;\n// if (totalAllocated != that.totalAllocated) return false;\n\n return true;\n }\n\n public int hashCode() {\n int result = (int) (hitCount ^ (hitCount >>> 32));\n result = 31 * result + (int) (missCount ^ (missCount >>> 32));\n result = 31 * result + (int) (size ^ (size >>> 32));\n// result = 31 * result + (int) (rehashCount ^ (rehashCount >>> 32));\n result = 31 * result + (int) (putAddCount ^ (putAddCount >>> 32));\n result = 31 * result + (int) (putReplaceCount ^ (putReplaceCount >>> 32));\n result = 31 * result + (int) (putFailCount ^ (putFailCount >>> 32));\n result = 31 * result + (int) (removeCount ^ (removeCount >>> 32));\n// result = 31 * result + (int) (totalAllocated ^ (totalAllocated >>> 32));\n return result;\n }\n}\n" }, { "path": "src/main/java/com/oath/halodb/Record.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport java.nio.ByteBuffer;\nimport java.util.Arrays;\nimport java.util.zip.CRC32;\n\npublic class Record {\n\n private final byte[] key, value;\n\n private InMemoryIndexMetaData recordMetaData;\n\n private Header header;\n\n public Record(byte[] key, byte[] value) {\n this.key = key;\n this.value = value;\n header = new Header(0, Versions.CURRENT_DATA_FILE_VERSION, (byte)key.length, value.length, -1);\n }\n\n ByteBuffer[] serialize() {\n ByteBuffer headerBuf = serializeHeaderAndComputeChecksum();\n return new ByteBuffer[] {headerBuf, ByteBuffer.wrap(key), ByteBuffer.wrap(value)};\n }\n\n static Record deserialize(ByteBuffer buffer, short keySize, int valueSize) {\n buffer.flip();\n byte[] key = new byte[keySize];\n byte[] value = new byte[valueSize];\n buffer.get(key);\n buffer.get(value);\n return new Record(key, value);\n }\n\n public byte[] getKey() {\n return key;\n }\n\n public byte[] getValue() {\n return value;\n }\n\n InMemoryIndexMetaData getRecordMetaData() {\n return recordMetaData;\n }\n\n void setRecordMetaData(InMemoryIndexMetaData recordMetaData) {\n this.recordMetaData = recordMetaData;\n }\n\n /**\n * @return recordSize which is HEADER_SIZE + key size + value size.\n */\n int getRecordSize() {\n return header.getRecordSize();\n }\n\n void setSequenceNumber(long sequenceNumber) {\n header.sequenceNumber = sequenceNumber;\n }\n\n long getSequenceNumber() {\n return header.getSequenceNumber();\n }\n\n void setVersion(int version) {\n if (version < 0 || version > 255) {\n throw new IllegalArgumentException(\"Got version \" + version + \". Record version must be in range [0,255]\");\n }\n header.version = version;\n }\n\n int getVersion() {\n return header.version;\n }\n\n Header getHeader() {\n return header;\n }\n\n void setHeader(Header header) {\n this.header = header;\n }\n\n private ByteBuffer serializeHeaderAndComputeChecksum() {\n ByteBuffer headerBuf = header.serialize();\n long checkSum = computeCheckSum(headerBuf.array());\n headerBuf.putInt(Header.CHECKSUM_OFFSET, Utils.toSignedIntFromLong(checkSum));\n return headerBuf;\n }\n\n boolean verifyChecksum() {\n ByteBuffer headerBuf = header.serialize();\n long checkSum = computeCheckSum(headerBuf.array());\n\n return checkSum == header.getCheckSum();\n }\n\n private long computeCheckSum(byte[] header) {\n CRC32 crc32 = new CRC32();\n\n // compute checksum with all but the first header element, key and value.\n crc32.update(header, Header.CHECKSUM_OFFSET + Header.CHECKSUM_SIZE, Header.HEADER_SIZE-Header.CHECKSUM_SIZE);\n crc32.update(key);\n crc32.update(value);\n return crc32.getValue();\n }\n\n @Override\n public boolean equals(Object obj) {\n // to be used in tests as we don't check if the headers are the same. \n\n if (this == obj) {\n return true;\n }\n if (!(obj instanceof Record)) {\n return false;\n }\n\n Record record = (Record)obj;\n return Arrays.equals(getKey(), record.getKey()) && Arrays.equals(getValue(), record.getValue());\n }\n\n static class Header {\n /**\n * crc - 4 bytes.\n * version - 1 byte.\n * key size - 1 bytes.\n * value size - 4 bytes.\n * sequence number - 8 bytes.\n */\n static final int CHECKSUM_OFFSET = 0;\n static final int VERSION_OFFSET = 4;\n static final int KEY_SIZE_OFFSET = 5;\n static final int VALUE_SIZE_OFFSET = 6;\n static final int SEQUENCE_NUMBER_OFFSET = 10;\n\n static final int HEADER_SIZE = 18;\n static final int CHECKSUM_SIZE = 4;\n\n private long checkSum;\n private int version;\n private byte keySize;\n private int valueSize;\n private long sequenceNumber;\n\n private int recordSize;\n\n Header(long checkSum, int version, byte keySize, int valueSize, long sequenceNumber) {\n this.checkSum = checkSum;\n this.version = version;\n this.keySize = keySize;\n this.valueSize = valueSize;\n this.sequenceNumber = sequenceNumber;\n recordSize = keySize + valueSize + HEADER_SIZE;\n }\n\n static Header deserialize(ByteBuffer buffer) {\n\n long checkSum = Utils.toUnsignedIntFromInt(buffer.getInt(CHECKSUM_OFFSET));\n int version = Utils.toUnsignedByte(buffer.get(VERSION_OFFSET));\n byte keySize = buffer.get(KEY_SIZE_OFFSET);\n int valueSize = buffer.getInt(VALUE_SIZE_OFFSET);\n long sequenceNumber = buffer.getLong(SEQUENCE_NUMBER_OFFSET);\n\n return new Header(checkSum, version, keySize, valueSize, sequenceNumber);\n }\n\n // checksum value can be computed only with record key and value. \n ByteBuffer serialize() {\n byte[] header = new byte[HEADER_SIZE];\n ByteBuffer headerBuffer = ByteBuffer.wrap(header);\n headerBuffer.put(VERSION_OFFSET, (byte)version);\n headerBuffer.put(KEY_SIZE_OFFSET, keySize);\n headerBuffer.putInt(VALUE_SIZE_OFFSET, valueSize);\n headerBuffer.putLong(SEQUENCE_NUMBER_OFFSET, sequenceNumber);\n\n return headerBuffer;\n }\n\n static boolean verifyHeader(Record.Header header) {\n return header.version >= 0 && header.version < 256\n && header.keySize > 0 && header.valueSize > 0\n && header.recordSize > 0 && header.sequenceNumber > 0;\n }\n\n byte getKeySize() {\n return keySize;\n }\n\n int getValueSize() {\n return valueSize;\n }\n\n int getRecordSize() {\n return recordSize;\n }\n\n long getSequenceNumber() {\n return sequenceNumber;\n }\n\n long getCheckSum() {\n return checkSum;\n }\n\n int getVersion() {\n return version;\n }\n }\n}\n" }, { "path": "src/main/java/com/oath/halodb/RecordKey.java", "content": "package com.oath.halodb;\n\nimport java.util.*;\n\npublic class RecordKey {\n final byte[] key;\n public RecordKey(byte[] key) {\n this.key = key;\n }\n\n public byte[] getBytes() {\n return key;\n }\n\n @Override\n public boolean equals(Object obj) {\n // to be used in tests as we don't check if the headers are the same.\n\n if (this == obj) {\n return true;\n }\n if (!(obj instanceof RecordKey)) {\n return false;\n }\n\n RecordKey recordKey = (RecordKey)obj;\n return Arrays.equals(this.key, recordKey.getBytes());\n }\n\n\n}\n" }, { "path": "src/main/java/com/oath/halodb/Segment.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\n// This code is a derivative work heavily modified from the OHC project. See NOTICE file for copyright and license.\n\npackage com.oath.halodb;\n\nimport com.oath.halodb.histo.EstimatedHistogram;\n\nimport java.util.concurrent.atomic.AtomicLongFieldUpdater;\n\nabstract class Segment {\n\n final HashTableValueSerializer valueSerializer;\n final int fixedValueLength;\n final int fixedKeyLength;\n\n private final Hasher hasher;\n\n private volatile long lock;\n private static final AtomicLongFieldUpdater lockFieldUpdater =\n AtomicLongFieldUpdater.newUpdater(Segment.class, \"lock\");\n\n Segment(HashTableValueSerializer valueSerializer, int fixedValueLength, Hasher hasher) {\n this(valueSerializer, fixedValueLength, -1, hasher);\n }\n\n Segment(HashTableValueSerializer valueSerializer, int fixedValueLength, int fixedKeyLength, Hasher hasher) {\n this.valueSerializer = valueSerializer;\n this.fixedValueLength = fixedValueLength;\n this.fixedKeyLength = fixedKeyLength;\n this.hasher = hasher;\n }\n\n\n\n boolean lock() {\n long t = Thread.currentThread().getId();\n\n if (t == lockFieldUpdater.get(this)) {\n return false;\n }\n while (true) {\n if (lockFieldUpdater.compareAndSet(this, 0L, t)) {\n return true;\n }\n\n // yield control to other thread.\n // Note: we cannot use LockSupport.parkNanos() as that does not\n // provide nanosecond resolution on Windows.\n Thread.yield();\n }\n }\n\n void unlock(boolean wasFirst) {\n if (!wasFirst) {\n return;\n }\n\n long t = Thread.currentThread().getId();\n boolean r = lockFieldUpdater.compareAndSet(this, t, 0L);\n assert r;\n }\n\n KeyBuffer keySource(byte[] key) {\n KeyBuffer keyBuffer = new KeyBuffer(key);\n return keyBuffer.finish(hasher);\n }\n\n abstract V getEntry(KeyBuffer key);\n\n abstract boolean containsEntry(KeyBuffer key);\n\n abstract boolean putEntry(byte[] key, V value, long hash, boolean ifAbsent, V oldValue);\n\n abstract boolean removeEntry(KeyBuffer key);\n\n abstract long size();\n\n abstract void release();\n\n abstract void clear();\n\n abstract long hitCount();\n\n abstract long missCount();\n\n abstract long putAddCount();\n\n abstract long putReplaceCount();\n\n abstract long removeCount();\n\n abstract void resetStatistics();\n\n abstract long rehashes();\n\n abstract float loadFactor();\n\n abstract int hashTableSize();\n\n abstract void updateBucketHistogram(EstimatedHistogram hist);\n\n\n //Used only in memory pool.\n\n long numberOfChunks() {\n return -1;\n }\n\n long numberOfSlots() {\n return -1;\n }\n\n long freeListSize() {\n return -1;\n }\n}\n" }, { "path": "src/main/java/com/oath/halodb/SegmentNonMemoryPool.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\n// This code is a derivative work heavily modified from the OHC project. See NOTICE file for copyright and license.\n\npackage com.oath.halodb;\n\nimport com.google.common.primitives.Ints;\nimport com.oath.halodb.histo.EstimatedHistogram;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nclass SegmentNonMemoryPool extends Segment {\n\n private static final Logger logger = LoggerFactory.getLogger(SegmentNonMemoryPool.class);\n\n // maximum hash table size\n private static final int MAX_TABLE_SIZE = 1 << 30;\n\n long size;\n Table table;\n\n private long hitCount;\n private long missCount;\n private long putAddCount;\n private long putReplaceCount;\n private long removeCount;\n\n private long threshold;\n private final float loadFactor;\n\n private long rehashes;\n long evictedEntries;\n\n private final HashAlgorithm hashAlgorithm;\n\n private static final boolean throwOOME = true;\n\n SegmentNonMemoryPool(OffHeapHashTableBuilder builder) {\n super(builder.getValueSerializer(), builder.getFixedValueSize(), builder.getHasher());\n\n this.hashAlgorithm = builder.getHashAlgorighm();\n\n int hts = builder.getHashTableSize();\n if (hts <= 0) {\n hts = 8192;\n }\n if (hts < 256) {\n hts = 256;\n }\n int msz = Ints.checkedCast(HashTableUtil.roundUpToPowerOf2(hts, MAX_TABLE_SIZE));\n table = Table.create(msz, throwOOME);\n if (table == null) {\n throw new RuntimeException(\"unable to allocate off-heap memory for segment\");\n }\n\n float lf = builder.getLoadFactor();\n if (lf <= .0d) {\n lf = .75f;\n }\n this.loadFactor = lf;\n threshold = (long) ((double) table.size() * loadFactor);\n }\n\n @Override\n void release() {\n boolean wasFirst = lock();\n try {\n table.release();\n table = null;\n } finally {\n unlock(wasFirst);\n }\n }\n\n @Override\n long size() {\n return size;\n }\n\n @Override\n long hitCount() {\n return hitCount;\n }\n\n @Override\n long missCount() {\n return missCount;\n }\n\n @Override\n long putAddCount() {\n return putAddCount;\n }\n\n @Override\n long putReplaceCount() {\n return putReplaceCount;\n }\n\n @Override\n long removeCount() {\n return removeCount;\n }\n\n @Override\n void resetStatistics() {\n rehashes = 0L;\n evictedEntries = 0L;\n hitCount = 0L;\n missCount = 0L;\n putAddCount = 0L;\n putReplaceCount = 0L;\n removeCount = 0L;\n }\n\n @Override\n long rehashes() {\n return rehashes;\n }\n\n @Override\n V getEntry(KeyBuffer key) {\n boolean wasFirst = lock();\n try {\n for (long hashEntryAdr = table.getFirst(key.hash());\n hashEntryAdr != 0L;\n hashEntryAdr = NonMemoryPoolHashEntries.getNext(hashEntryAdr)) {\n\n if (key.sameKey(hashEntryAdr)) {\n hitCount++;\n return valueSerializer.deserialize(Uns.readOnlyBuffer(hashEntryAdr, fixedValueLength, NonMemoryPoolHashEntries.ENTRY_OFF_DATA + NonMemoryPoolHashEntries.getKeyLen(hashEntryAdr)));\n }\n }\n\n missCount++;\n return null;\n } finally {\n unlock(wasFirst);\n }\n }\n\n @Override\n boolean containsEntry(KeyBuffer key) {\n boolean wasFirst = lock();\n try {\n for (long hashEntryAdr = table.getFirst(key.hash());\n hashEntryAdr != 0L;\n hashEntryAdr = NonMemoryPoolHashEntries.getNext(hashEntryAdr)) {\n if (key.sameKey(hashEntryAdr)) {\n hitCount++;\n return true;\n }\n }\n\n missCount++;\n return false;\n } finally {\n unlock(wasFirst);\n }\n }\n\n @Override\n boolean putEntry(byte[] key, V value, long hash, boolean ifAbsent, V oldValue) {\n long oldValueAdr = 0L;\n try {\n if (oldValue != null) {\n oldValueAdr = Uns.allocate(fixedValueLength, throwOOME);\n if (oldValueAdr == 0L) {\n throw new RuntimeException(\"Unable to allocate \" + fixedValueLength + \" bytes in off-heap\");\n }\n valueSerializer.serialize(oldValue, Uns.directBufferFor(oldValueAdr, 0, fixedValueLength, false));\n }\n\n long hashEntryAdr;\n if ((hashEntryAdr = Uns.allocate(HashTableUtil.allocLen(key.length, fixedValueLength), throwOOME)) == 0L) {\n // entry too large to be inserted or OS is not able to provide enough memory\n removeEntry(keySource(key));\n return false;\n }\n\n // initialize hash entry\n NonMemoryPoolHashEntries.init(key.length, hashEntryAdr);\n serializeForPut(key, value, hashEntryAdr);\n\n if (putEntry(hashEntryAdr, hash, key.length, ifAbsent, oldValueAdr)) {\n return true;\n }\n\n Uns.free(hashEntryAdr);\n return false;\n } finally {\n Uns.free(oldValueAdr);\n }\n }\n\n private boolean putEntry(long newHashEntryAdr, long hash, long keyLen, boolean putIfAbsent, long oldValueAddr) {\n long removeHashEntryAdr = 0L;\n boolean wasFirst = lock();\n try {\n long hashEntryAdr;\n long prevEntryAdr = 0L;\n for (hashEntryAdr = table.getFirst(hash);\n hashEntryAdr != 0L;\n prevEntryAdr = hashEntryAdr, hashEntryAdr = NonMemoryPoolHashEntries.getNext(hashEntryAdr)) {\n if (notSameKey(newHashEntryAdr, hash, keyLen, hashEntryAdr)) {\n continue;\n }\n\n // putIfAbsent is true, but key is already present, return.\n if (putIfAbsent) {\n return false;\n }\n\n // key already exists, we just need to replace the value.\n if (oldValueAddr != 0L) {\n // code for replace() operation\n if (!Uns.memoryCompare(hashEntryAdr, NonMemoryPoolHashEntries.ENTRY_OFF_DATA + keyLen, oldValueAddr, 0L, fixedValueLength)) {\n return false;\n }\n }\n\n removeInternal(hashEntryAdr, prevEntryAdr, hash);\n removeHashEntryAdr = hashEntryAdr;\n\n break;\n }\n\n // key is not present in the map, therefore we need to add a new entry.\n if (hashEntryAdr == 0L) {\n\n // key is not present but old value is not null.\n // we consider this as a mismatch and return.\n if (oldValueAddr != 0) {\n return false;\n }\n\n if (size >= threshold) {\n rehash();\n }\n\n size++;\n }\n\n add(newHashEntryAdr, hash);\n\n if (hashEntryAdr == 0L) {\n putAddCount++;\n } else {\n putReplaceCount++;\n }\n\n return true;\n } finally {\n unlock(wasFirst);\n if (removeHashEntryAdr != 0L) {\n Uns.free(removeHashEntryAdr);\n }\n }\n }\n\n private static boolean notSameKey(long newHashEntryAdr, long newHash, long newKeyLen, long hashEntryAdr) {\n long serKeyLen = NonMemoryPoolHashEntries.getKeyLen(hashEntryAdr);\n return serKeyLen != newKeyLen\n || !Uns.memoryCompare(hashEntryAdr, NonMemoryPoolHashEntries.ENTRY_OFF_DATA, newHashEntryAdr, NonMemoryPoolHashEntries.ENTRY_OFF_DATA, serKeyLen);\n }\n\n private void serializeForPut(byte[] key, V value, long hashEntryAdr) {\n try {\n Uns.buffer(hashEntryAdr, key.length, NonMemoryPoolHashEntries.ENTRY_OFF_DATA).put(key);\n if (value != null) {\n valueSerializer.serialize(value, Uns.buffer(hashEntryAdr, fixedValueLength, NonMemoryPoolHashEntries.ENTRY_OFF_DATA + key.length));\n }\n } catch (Throwable e) {\n freeAndThrow(e, hashEntryAdr);\n }\n }\n\n private void freeAndThrow(Throwable e, long hashEntryAdr) {\n Uns.free(hashEntryAdr);\n if (e instanceof RuntimeException) {\n throw (RuntimeException) e;\n }\n if (e instanceof Error) {\n throw (Error) e;\n }\n throw new RuntimeException(e);\n }\n\n @Override\n void clear() {\n boolean wasFirst = lock();\n try {\n size = 0L;\n\n long next;\n for (int p = 0; p < table.size(); p++) {\n for (long hashEntryAdr = table.getFirst(p);\n hashEntryAdr != 0L;\n hashEntryAdr = next) {\n next = NonMemoryPoolHashEntries.getNext(hashEntryAdr);\n Uns.free(hashEntryAdr);\n }\n }\n\n table.clear();\n } finally {\n unlock(wasFirst);\n }\n }\n\n @Override\n boolean removeEntry(KeyBuffer key) {\n long removeHashEntryAdr = 0L;\n boolean wasFirst = lock();\n try {\n long prevEntryAdr = 0L;\n for (long hashEntryAdr = table.getFirst(key.hash());\n hashEntryAdr != 0L;\n prevEntryAdr = hashEntryAdr, hashEntryAdr = NonMemoryPoolHashEntries.getNext(hashEntryAdr)) {\n if (!key.sameKey(hashEntryAdr)) {\n continue;\n }\n\n // remove existing entry\n\n removeHashEntryAdr = hashEntryAdr;\n removeInternal(hashEntryAdr, prevEntryAdr, key.hash());\n\n size--;\n removeCount++;\n\n return true;\n }\n\n return false;\n } finally {\n unlock(wasFirst);\n if (removeHashEntryAdr != 0L) {\n Uns.free(removeHashEntryAdr);\n }\n }\n }\n\n private void rehash() {\n long start = System.currentTimeMillis();\n Table tab = table;\n int tableSize = tab.size();\n if (tableSize > MAX_TABLE_SIZE) {\n // already at max hash table size\n return;\n }\n\n Table newTable = Table.create(tableSize * 2, throwOOME);\n if (newTable == null) {\n return;\n }\n long next;\n\n Hasher hasher = Hasher.create(hashAlgorithm);\n\n for (int part = 0; part < tableSize; part++) {\n for (long hashEntryAdr = tab.getFirst(part);\n hashEntryAdr != 0L;\n hashEntryAdr = next) {\n\n next = NonMemoryPoolHashEntries.getNext(hashEntryAdr);\n NonMemoryPoolHashEntries.setNext(hashEntryAdr, 0L);\n long hash = hasher.hash(hashEntryAdr, NonMemoryPoolHashEntries.ENTRY_OFF_DATA, NonMemoryPoolHashEntries.getKeyLen(hashEntryAdr));\n newTable.addAsHead(hash, hashEntryAdr);\n }\n }\n\n threshold = (long) ((float) newTable.size() * loadFactor);\n table.release();\n table = newTable;\n rehashes++;\n logger.info(\"Completed rehashing segment in {} ms.\", (System.currentTimeMillis() - start));\n }\n\n float loadFactor() {\n return loadFactor;\n }\n\n int hashTableSize() {\n return table.size();\n }\n\n void updateBucketHistogram(EstimatedHistogram hist) {\n boolean wasFirst = lock();\n try {\n table.updateBucketHistogram(hist);\n } finally {\n unlock(wasFirst);\n }\n }\n\n void getEntryAddresses(int mapSegmentIndex, int nSegments, LongArrayList hashEntryAdrs) {\n boolean wasFirst = lock();\n try {\n for (; nSegments-- > 0 && mapSegmentIndex < table.size(); mapSegmentIndex++) {\n for (long hashEntryAdr = table.getFirst(mapSegmentIndex);\n hashEntryAdr != 0L;\n hashEntryAdr = NonMemoryPoolHashEntries.getNext(hashEntryAdr)) {\n hashEntryAdrs.add(hashEntryAdr);\n }\n }\n } finally {\n unlock(wasFirst);\n }\n }\n\n static final class Table {\n\n final int mask;\n final long address;\n private boolean released;\n\n static Table create(int hashTableSize, boolean throwOOME) {\n int msz = Ints.checkedCast(HashTableUtil.NON_MEMORY_POOL_BUCKET_ENTRY_LEN * hashTableSize);\n long address = Uns.allocate(msz, throwOOME);\n return address != 0L ? new Table(address, hashTableSize) : null;\n }\n\n private Table(long address, int hashTableSize) {\n this.address = address;\n this.mask = hashTableSize - 1;\n clear();\n }\n\n void clear() {\n // It's important to initialize the hash table memory.\n // (uninitialized memory will cause problems - endless loops, JVM crashes, damaged data, etc)\n Uns.setMemory(address, 0L, HashTableUtil.NON_MEMORY_POOL_BUCKET_ENTRY_LEN * size(), (byte) 0);\n }\n\n void release() {\n Uns.free(address);\n released = true;\n }\n\n protected void finalize() throws Throwable {\n if (!released) {\n Uns.free(address);\n }\n super.finalize();\n }\n\n long getFirst(long hash) {\n return Uns.getLong(address, bucketOffset(hash));\n }\n\n void setFirst(long hash, long hashEntryAdr) {\n Uns.putLong(address, bucketOffset(hash), hashEntryAdr);\n }\n\n long bucketOffset(long hash) {\n return bucketIndexForHash(hash) * HashTableUtil.NON_MEMORY_POOL_BUCKET_ENTRY_LEN;\n }\n\n private int bucketIndexForHash(long hash) {\n return (int) (hash & mask);\n }\n\n void removeLink(long hash, long hashEntryAdr, long prevEntryAdr) {\n long next = NonMemoryPoolHashEntries.getNext(hashEntryAdr);\n\n removeLinkInternal(hash, hashEntryAdr, prevEntryAdr, next);\n }\n\n void replaceSentinelLink(long hash, long hashEntryAdr, long prevEntryAdr, long newHashEntryAdr) {\n NonMemoryPoolHashEntries.setNext(newHashEntryAdr, NonMemoryPoolHashEntries.getNext(hashEntryAdr));\n\n removeLinkInternal(hash, hashEntryAdr, prevEntryAdr, newHashEntryAdr);\n }\n\n private void removeLinkInternal(long hash, long hashEntryAdr, long prevEntryAdr, long next) {\n long head = getFirst(hash);\n if (head == hashEntryAdr) {\n setFirst(hash, next);\n } else if (prevEntryAdr != 0L) {\n if (prevEntryAdr == -1L) {\n for (long adr = head;\n adr != 0L;\n prevEntryAdr = adr, adr = NonMemoryPoolHashEntries.getNext(adr)) {\n if (adr == hashEntryAdr) {\n break;\n }\n }\n }\n NonMemoryPoolHashEntries.setNext(prevEntryAdr, next);\n }\n }\n\n void addAsHead(long hash, long hashEntryAdr) {\n long head = getFirst(hash);\n NonMemoryPoolHashEntries.setNext(hashEntryAdr, head);\n setFirst(hash, hashEntryAdr);\n }\n\n int size() {\n return mask + 1;\n }\n\n void updateBucketHistogram(EstimatedHistogram h) {\n for (int i = 0; i < size(); i++) {\n int len = 0;\n for (long adr = getFirst(i); adr != 0L; adr = NonMemoryPoolHashEntries.getNext(adr)) {\n len++;\n }\n h.add(len + 1);\n }\n }\n }\n\n private void removeInternal(long hashEntryAdr, long prevEntryAdr, long hash) {\n table.removeLink(hash, hashEntryAdr, prevEntryAdr);\n }\n\n private void add(long hashEntryAdr, long hash) {\n table.addAsHead(hash, hashEntryAdr);\n }\n\n @Override\n public String toString() {\n return String.valueOf(size);\n }\n}\n" }, { "path": "src/main/java/com/oath/halodb/SegmentStats.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport com.google.common.base.MoreObjects;\n\nclass SegmentStats {\n\n private final long noOfEntries;\n private final long numberOfChunks;\n private final long numberOfSlots;\n private final long freeListSize;\n\n public SegmentStats(long noOfEntries, long numberOfChunks, long numberOfSlots, long freeListSize) {\n this.noOfEntries = noOfEntries;\n this.numberOfChunks = numberOfChunks;\n this.numberOfSlots = numberOfSlots;\n this.freeListSize = freeListSize;\n }\n\n @Override\n public String toString() {\n MoreObjects.ToStringHelper helper =\n MoreObjects.toStringHelper(\"\").add(\"noOfEntries\", this.noOfEntries);\n\n // all these values will be -1 for non-memory pool, hence ignore. \n if (numberOfChunks != -1) {\n helper.add(\"numberOfChunks\", numberOfChunks);\n }\n if (numberOfSlots != -1) {\n helper.add(\"numberOfSlots\", numberOfSlots);\n }\n if (freeListSize != -1) {\n helper.add(\"freeListSize\", freeListSize);\n }\n return helper.toString();\n }\n\n @Override\n public boolean equals(Object obj) {\n if (obj == this)\n return true;\n\n if (!(obj instanceof SegmentStats))\n return false;\n\n SegmentStats that = (SegmentStats) obj;\n return that.noOfEntries == noOfEntries\n && that.numberOfChunks == numberOfChunks\n && that.numberOfSlots == numberOfSlots\n && that.freeListSize == freeListSize;\n }\n\n @Override\n public int hashCode() {\n int result = 1;\n result = 31 * result + Long.hashCode(noOfEntries);\n result = 31 * result + Long.hashCode(numberOfChunks);\n result = 31 * result + Long.hashCode(numberOfSlots);\n result = 31 * result + Long.hashCode(freeListSize);\n\n return result;\n }\n}\n" }, { "path": "src/main/java/com/oath/halodb/SegmentWithMemoryPool.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.primitives.Ints;\nimport com.oath.halodb.histo.EstimatedHistogram;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport java.nio.ByteBuffer;\nimport java.util.ArrayList;\nimport java.util.List;\n\nclass SegmentWithMemoryPool extends Segment {\n\n private static final Logger logger = LoggerFactory.getLogger(SegmentWithMemoryPool.class);\n\n // maximum hash table size\n private static final int MAX_TABLE_SIZE = 1 << 30;\n\n private long hitCount = 0;\n private long size = 0;\n private long missCount = 0;\n private long putAddCount = 0;\n private long putReplaceCount = 0;\n private long removeCount = 0;\n private long threshold = 0;\n private final float loadFactor;\n private long rehashes = 0;\n\n private final List chunks;\n private byte currentChunkIndex = -1;\n\n private final int chunkSize;\n\n private final MemoryPoolAddress emptyAddress = new MemoryPoolAddress((byte) -1, -1);\n\n private MemoryPoolAddress freeListHead = emptyAddress;\n private long freeListSize = 0;\n\n private final int fixedSlotSize;\n\n private final HashTableValueSerializer valueSerializer;\n\n private Table table;\n\n private final ByteBuffer oldValueBuffer = ByteBuffer.allocate(fixedValueLength);\n private final ByteBuffer newValueBuffer = ByteBuffer.allocate(fixedValueLength);\n\n private final HashAlgorithm hashAlgorithm;\n\n SegmentWithMemoryPool(OffHeapHashTableBuilder builder) {\n super(builder.getValueSerializer(), builder.getFixedValueSize(), builder.getFixedKeySize(),\n builder.getHasher());\n\n this.chunks = new ArrayList<>();\n this.chunkSize = builder.getMemoryPoolChunkSize();\n this.valueSerializer = builder.getValueSerializer();\n this.fixedSlotSize = MemoryPoolHashEntries.HEADER_SIZE + fixedKeyLength + fixedValueLength;\n this.hashAlgorithm = builder.getHashAlgorighm();\n\n int hts = builder.getHashTableSize();\n if (hts <= 0) {\n hts = 8192;\n }\n if (hts < 256) {\n hts = 256;\n }\n int msz = Ints.checkedCast(HashTableUtil.roundUpToPowerOf2(hts, MAX_TABLE_SIZE));\n table = Table.create(msz);\n if (table == null) {\n throw new RuntimeException(\"unable to allocate off-heap memory for segment\");\n }\n\n float lf = builder.getLoadFactor();\n if (lf <= .0d) {\n lf = .75f;\n }\n this.loadFactor = lf;\n threshold = (long) ((double) table.size() * loadFactor);\n }\n\n @Override\n public V getEntry(KeyBuffer key) {\n boolean wasFirst = lock();\n try {\n for (MemoryPoolAddress address = table.getFirst(key.hash());\n address.chunkIndex >= 0;\n address = getNext(address)) {\n\n MemoryPoolChunk chunk = chunks.get(address.chunkIndex);\n if (chunk.compareKey(address.chunkOffset, key.buffer)) {\n hitCount++;\n return valueSerializer.deserialize(chunk.readOnlyValueByteBuffer(address.chunkOffset));\n }\n }\n\n missCount++;\n return null;\n } finally {\n unlock(wasFirst);\n }\n }\n\n @Override\n public boolean containsEntry(KeyBuffer key) {\n boolean wasFirst = lock();\n try {\n for (MemoryPoolAddress address = table.getFirst(key.hash());\n address.chunkIndex >= 0;\n address = getNext(address)) {\n\n MemoryPoolChunk chunk = chunks.get(address.chunkIndex);\n if (chunk.compareKey(address.chunkOffset, key.buffer)) {\n hitCount++;\n return true;\n }\n }\n\n missCount++;\n return false;\n } finally {\n unlock(wasFirst);\n }\n }\n\n @Override\n boolean putEntry(byte[] key, V value, long hash, boolean putIfAbsent, V oldValue) {\n boolean wasFirst = lock();\n try {\n if (oldValue != null) {\n oldValueBuffer.clear();\n valueSerializer.serialize(oldValue, oldValueBuffer);\n }\n newValueBuffer.clear();\n valueSerializer.serialize(value, newValueBuffer);\n\n MemoryPoolAddress first = table.getFirst(hash);\n for (MemoryPoolAddress address = first; address.chunkIndex >= 0; address = getNext(address)) {\n MemoryPoolChunk chunk = chunks.get(address.chunkIndex);\n if (chunk.compareKey(address.chunkOffset, key)) {\n // key is already present in the segment. \n\n // putIfAbsent is true, but key is already present, return.\n if (putIfAbsent) {\n return false;\n }\n\n // code for replace() operation\n if (oldValue != null) {\n if (!chunk.compareValue(address.chunkOffset, oldValueBuffer.array())) {\n return false;\n }\n }\n\n // replace value with the new one.\n chunk.setValue(newValueBuffer.array(), address.chunkOffset);\n putReplaceCount++;\n return true;\n }\n }\n\n if (oldValue != null) {\n // key is not present but old value is not null.\n // we consider this as a mismatch and return.\n return false;\n }\n\n if (size >= threshold) {\n rehash();\n first = table.getFirst(hash);\n }\n\n // key is not present in the segment, we need to add a new entry.\n MemoryPoolAddress nextSlot = writeToFreeSlot(key, newValueBuffer.array(), first);\n table.addAsHead(hash, nextSlot);\n size++;\n putAddCount++;\n } finally {\n unlock(wasFirst);\n }\n\n return true;\n }\n\n @Override\n public boolean removeEntry(KeyBuffer key) {\n boolean wasFirst = lock();\n try {\n MemoryPoolAddress previous = null;\n for (MemoryPoolAddress address = table.getFirst(key.hash());\n address.chunkIndex >= 0;\n previous = address, address = getNext(address)) {\n\n MemoryPoolChunk chunk = chunks.get(address.chunkIndex);\n if (chunk.compareKey(address.chunkOffset, key.buffer)) {\n removeInternal(address, previous, key.hash());\n removeCount++;\n size--;\n return true;\n }\n }\n\n return false;\n } finally {\n unlock(wasFirst);\n }\n }\n\n private MemoryPoolAddress getNext(MemoryPoolAddress address) {\n if (address.chunkIndex < 0 || address.chunkIndex >= chunks.size()) {\n throw new IllegalArgumentException(\"Invalid chunk index \" + address.chunkIndex + \". Chunk size \" + chunks.size());\n }\n\n MemoryPoolChunk chunk = chunks.get(address.chunkIndex);\n return chunk.getNextAddress(address.chunkOffset);\n }\n\n private MemoryPoolAddress writeToFreeSlot(byte[] key, byte[] value, MemoryPoolAddress nextAddress) {\n if (!freeListHead.equals(emptyAddress)) {\n // write to the head of the free list.\n MemoryPoolAddress temp = freeListHead;\n freeListHead = chunks.get(freeListHead.chunkIndex).getNextAddress(freeListHead.chunkOffset);\n chunks.get(temp.chunkIndex).fillSlot(temp.chunkOffset, key, value, nextAddress);\n --freeListSize;\n return temp;\n }\n\n if (currentChunkIndex == -1 || chunks.get(currentChunkIndex).remaining() < fixedSlotSize) {\n if (chunks.size() > Byte.MAX_VALUE) {\n logger.error(\"No more memory left. Each segment can have at most {} chunks.\", Byte.MAX_VALUE + 1);\n throw new OutOfMemoryError(\"Each segment can have at most \" + (Byte.MAX_VALUE + 1) + \" chunks.\");\n }\n\n // There is no chunk allocated for this segment or the current chunk being written to has no space left.\n // allocate an new one. \n chunks.add(MemoryPoolChunk.create(chunkSize, fixedKeyLength, fixedValueLength));\n ++currentChunkIndex;\n }\n\n MemoryPoolChunk currentWriteChunk = chunks.get(currentChunkIndex);\n MemoryPoolAddress slotAddress = new MemoryPoolAddress(currentChunkIndex, currentWriteChunk.getWriteOffset());\n currentWriteChunk.fillNextSlot(key, value, nextAddress);\n return slotAddress;\n }\n\n private void removeInternal(MemoryPoolAddress address, MemoryPoolAddress previous, long hash) {\n MemoryPoolAddress next = chunks.get(address.chunkIndex).getNextAddress(address.chunkOffset);\n if (table.getFirst(hash).equals(address)) {\n table.addAsHead(hash, next);\n } else if (previous == null) {\n //this should never happen. \n throw new IllegalArgumentException(\"Removing entry which is not head but with previous null\");\n } else {\n chunks.get(previous.chunkIndex).setNextAddress(previous.chunkOffset, next);\n }\n\n chunks.get(address.chunkIndex).setNextAddress(address.chunkOffset, freeListHead);\n freeListHead = address;\n ++freeListSize;\n }\n\n private void rehash() {\n long start = System.currentTimeMillis();\n Table currentTable = table;\n int tableSize = currentTable.size();\n if (tableSize > MAX_TABLE_SIZE) {\n return;\n }\n\n Table newTable = Table.create(tableSize * 2);\n Hasher hasher = Hasher.create(hashAlgorithm);\n MemoryPoolAddress next;\n\n for (int i = 0; i < tableSize; i++) {\n for (MemoryPoolAddress address = table.getFirst(i); address.chunkIndex >= 0; address = next) {\n long hash = chunks.get(address.chunkIndex).computeHash(address.chunkOffset, hasher);\n next = getNext(address);\n MemoryPoolAddress first = newTable.getFirst(hash);\n newTable.addAsHead(hash, address);\n chunks.get(address.chunkIndex).setNextAddress(address.chunkOffset, first);\n }\n }\n\n threshold = (long) ((float) newTable.size() * loadFactor);\n table.release();\n table = newTable;\n rehashes++;\n\n logger.info(\"Completed rehashing segment in {} ms.\", (System.currentTimeMillis() - start));\n }\n\n @Override\n long size() {\n return size;\n }\n\n @Override\n void release() {\n boolean wasFirst = lock();\n try {\n chunks.forEach(MemoryPoolChunk::destroy);\n chunks.clear();\n currentChunkIndex = -1;\n size = 0;\n table.release();\n } finally {\n unlock(wasFirst);\n }\n\n }\n\n @Override\n void clear() {\n boolean wasFirst = lock();\n try {\n chunks.forEach(MemoryPoolChunk::destroy);\n chunks.clear();\n currentChunkIndex = -1;\n size = 0;\n table.clear();\n } finally {\n unlock(wasFirst);\n }\n }\n\n @Override\n long hitCount() {\n return hitCount;\n }\n\n @Override\n long missCount() {\n return missCount;\n }\n\n @Override\n long putAddCount() {\n return putAddCount;\n }\n\n @Override\n long putReplaceCount() {\n return putReplaceCount;\n }\n\n @Override\n long removeCount() {\n return removeCount;\n }\n\n @Override\n void resetStatistics() {\n rehashes = 0L;\n hitCount = 0L;\n missCount = 0L;\n putAddCount = 0L;\n putReplaceCount = 0L;\n removeCount = 0L;\n }\n\n @Override\n long numberOfChunks() {\n return chunks.size();\n }\n\n @Override\n long numberOfSlots() {\n return chunks.size() * chunkSize / fixedSlotSize;\n }\n\n @Override\n long freeListSize() {\n return freeListSize;\n }\n\n @Override\n long rehashes() {\n return rehashes;\n }\n\n @Override\n float loadFactor() {\n return loadFactor;\n }\n\n @Override\n int hashTableSize() {\n return table.size();\n }\n\n @Override\n void updateBucketHistogram(EstimatedHistogram hist) {\n boolean wasFirst = lock();\n try {\n table.updateBucketHistogram(hist, chunks);\n } finally {\n unlock(wasFirst);\n }\n }\n\n static final class Table {\n\n final int mask;\n final long address;\n private boolean released;\n\n static Table create(int hashTableSize) {\n int msz = Ints.checkedCast(HashTableUtil.MEMORY_POOL_BUCKET_ENTRY_LEN * hashTableSize);\n long address = Uns.allocate(msz, true);\n return address != 0L ? new Table(address, hashTableSize) : null;\n }\n\n private Table(long address, int hashTableSize) {\n this.address = address;\n this.mask = hashTableSize - 1;\n clear();\n }\n\n void clear() {\n Uns.setMemory(address, 0L, HashTableUtil.MEMORY_POOL_BUCKET_ENTRY_LEN * size(), (byte) -1);\n }\n\n void release() {\n Uns.free(address);\n released = true;\n }\n\n protected void finalize() throws Throwable {\n if (!released) {\n Uns.free(address);\n }\n super.finalize();\n }\n\n MemoryPoolAddress getFirst(long hash) {\n long bOffset = address + bucketOffset(hash);\n byte chunkIndex = Uns.getByte(bOffset, 0);\n int chunkOffset = Uns.getInt(bOffset, 1);\n return new MemoryPoolAddress(chunkIndex, chunkOffset);\n\n }\n\n void addAsHead(long hash, MemoryPoolAddress entryAddress) {\n long bOffset = address + bucketOffset(hash);\n Uns.putByte(bOffset, 0, entryAddress.chunkIndex);\n Uns.putInt(bOffset, 1, entryAddress.chunkOffset);\n }\n\n long bucketOffset(long hash) {\n return bucketIndexForHash(hash) * HashTableUtil.MEMORY_POOL_BUCKET_ENTRY_LEN;\n }\n\n private int bucketIndexForHash(long hash) {\n return (int) (hash & mask);\n }\n\n int size() {\n return mask + 1;\n }\n\n void updateBucketHistogram(EstimatedHistogram h, final List chunks) {\n for (int i = 0; i < size(); i++) {\n int len = 0;\n for (MemoryPoolAddress adr = getFirst(i); adr.chunkIndex >= 0;\n adr = chunks.get(adr.chunkIndex).getNextAddress(adr.chunkOffset)) {\n len++;\n }\n h.add(len + 1);\n }\n }\n }\n\n @VisibleForTesting\n MemoryPoolAddress getFreeListHead() {\n return freeListHead;\n }\n\n @VisibleForTesting\n int getChunkWriteOffset(int index) {\n return chunks.get(index).getWriteOffset();\n }\n}\n" }, { "path": "src/main/java/com/oath/halodb/TombstoneEntry.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport java.nio.ByteBuffer;\nimport java.util.zip.CRC32;\n\nclass TombstoneEntry {\n //TODO: test.\n\n /**\n * crc - 4 byte\n * version - 1 byte\n * Key size - 1 byte\n * Sequence number - 8 byte\n */\n static final int TOMBSTONE_ENTRY_HEADER_SIZE = 4 + 1 + 1 + 8;\n static final int CHECKSUM_SIZE = 4;\n\n static final int CHECKSUM_OFFSET = 0;\n static final int VERSION_OFFSET = 4;\n static final int SEQUENCE_NUMBER_OFFSET = 5;\n static final int KEY_SIZE_OFFSET = 13;\n\n private final byte[] key;\n private final long sequenceNumber;\n private final long checkSum;\n private final int version;\n\n TombstoneEntry(byte[] key, long sequenceNumber, long checkSum, int version) {\n this.key = key;\n this.sequenceNumber = sequenceNumber;\n this.checkSum = checkSum;\n this.version = version;\n }\n\n byte[] getKey() {\n return key;\n }\n\n long getSequenceNumber() {\n return sequenceNumber;\n }\n\n int getVersion() {\n return version;\n }\n\n long getCheckSum() {\n return checkSum;\n }\n\n int size() {\n return TOMBSTONE_ENTRY_HEADER_SIZE + key.length;\n }\n\n ByteBuffer[] serialize() {\n byte keySize = (byte)key.length;\n ByteBuffer header = ByteBuffer.allocate(TOMBSTONE_ENTRY_HEADER_SIZE);\n header.put(VERSION_OFFSET, (byte)version);\n header.putLong(SEQUENCE_NUMBER_OFFSET, sequenceNumber);\n header.put(KEY_SIZE_OFFSET, keySize);\n long crc32 = computeCheckSum(header.array());\n header.putInt(CHECKSUM_OFFSET, Utils.toSignedIntFromLong(crc32));\n return new ByteBuffer[] {header, ByteBuffer.wrap(key)};\n }\n\n static TombstoneEntry deserialize(ByteBuffer buffer) {\n long crc32 = Utils.toUnsignedIntFromInt(buffer.getInt());\n int version = Utils.toUnsignedByte(buffer.get());\n long sequenceNumber = buffer.getLong();\n int keySize = (int)buffer.get();\n byte[] key = new byte[keySize];\n buffer.get(key);\n\n return new TombstoneEntry(key, sequenceNumber, crc32, version);\n }\n\n // returns null if a corrupted entry is detected. \n static TombstoneEntry deserializeIfNotCorrupted(ByteBuffer buffer) {\n if (buffer.remaining() < TOMBSTONE_ENTRY_HEADER_SIZE) {\n return null;\n }\n\n long crc32 = Utils.toUnsignedIntFromInt(buffer.getInt());\n int version = Utils.toUnsignedByte(buffer.get());\n long sequenceNumber = buffer.getLong();\n int keySize = (int)buffer.get();\n if (sequenceNumber < 0 || keySize <= 0 || version < 0 || version > 255 || buffer.remaining() < keySize)\n return null;\n\n byte[] key = new byte[keySize];\n buffer.get(key);\n\n TombstoneEntry entry = new TombstoneEntry(key, sequenceNumber, crc32, version);\n if (entry.computeCheckSum() != entry.checkSum) {\n return null;\n }\n\n return entry;\n }\n\n private long computeCheckSum(byte[] header) {\n CRC32 crc32 = new CRC32();\n crc32.update(header, CHECKSUM_OFFSET + CHECKSUM_SIZE, TOMBSTONE_ENTRY_HEADER_SIZE - CHECKSUM_SIZE);\n crc32.update(key);\n return crc32.getValue();\n }\n\n long computeCheckSum() {\n ByteBuffer header = ByteBuffer.allocate(TOMBSTONE_ENTRY_HEADER_SIZE);\n header.put(VERSION_OFFSET, (byte)version);\n header.putLong(SEQUENCE_NUMBER_OFFSET, sequenceNumber);\n header.put(KEY_SIZE_OFFSET, (byte)key.length);\n return computeCheckSum(header.array());\n }\n}\n" }, { "path": "src/main/java/com/oath/halodb/TombstoneFile.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport java.io.File;\nimport java.io.IOException;\nimport java.io.RandomAccessFile;\nimport java.nio.ByteBuffer;\nimport java.nio.channels.FileChannel;\nimport java.nio.file.Files;\nimport java.nio.file.Path;\nimport java.util.Iterator;\nimport java.util.Objects;\n\nimport static java.nio.file.StandardCopyOption.ATOMIC_MOVE;\nimport static java.nio.file.StandardCopyOption.REPLACE_EXISTING;\n\nclass TombstoneFile {\n private static final Logger logger = LoggerFactory.getLogger(TombstoneFile.class);\n\n private final File backingFile;\n private FileChannel channel;\n private final DBDirectory dbDirectory;\n\n private final HaloDBOptions options;\n\n private long unFlushedData = 0;\n private long writeOffset = 0;\n\n static final String TOMBSTONE_FILE_NAME = \".tombstone\";\n private static final String nullMessage = \"Tombstone entry cannot be null\";\n\n static TombstoneFile create(DBDirectory dbDirectory, int fileId, HaloDBOptions options) throws IOException {\n File file = getTombstoneFile(dbDirectory, fileId);\n\n while (!file.createNewFile()) {\n // file already exists try another one.\n fileId++;\n file = getTombstoneFile(dbDirectory, fileId);\n }\n\n TombstoneFile tombstoneFile = new TombstoneFile(file, options, dbDirectory);\n tombstoneFile.open();\n\n return tombstoneFile;\n }\n\n TombstoneFile(File backingFile, HaloDBOptions options, DBDirectory dbDirectory) {\n this.backingFile = backingFile;\n this.options = options;\n this.dbDirectory = dbDirectory;\n }\n\n void open() throws IOException {\n channel = new RandomAccessFile(backingFile, \"rw\").getChannel();\n }\n\n void close() throws IOException {\n if (channel != null) {\n channel.close();\n }\n }\n\n void delete() throws IOException {\n close();\n if (backingFile != null) {\n backingFile.delete();\n }\n }\n\n void write(TombstoneEntry entry) throws IOException {\n Objects.requireNonNull(entry, nullMessage);\n\n ByteBuffer[] contents = entry.serialize();\n long toWrite = 0;\n for (ByteBuffer buffer : contents) {\n toWrite += buffer.remaining();\n }\n long written = 0;\n while (written < toWrite) {\n written += channel.write(contents);\n }\n\n writeOffset += written;\n unFlushedData += written;\n if (options.isSyncWrite() || (options.getFlushDataSizeBytes() != -1 && unFlushedData > options.getFlushDataSizeBytes())) {\n flushToDisk();\n unFlushedData = 0;\n }\n }\n\n long getWriteOffset() {\n return writeOffset;\n }\n\n void flushToDisk() throws IOException {\n if (channel != null && channel.isOpen())\n channel.force(true);\n }\n\n /**\n * Copies to a temp file those entries whose computed checksum matches the stored one and then\n * atomically rename the temp file to the current file.\n * Records in the file which occur after a corrupted record are discarded.\n * Current file is deleted after copy.\n * This method is called if we detect an unclean shutdown.\n */\n TombstoneFile repairFile(DBDirectory dbDirectory) throws IOException {\n TombstoneFile repairFile = createRepairFile();\n\n logger.info(\"Repairing tombstone file {}. Records with the correct checksum will be copied to {}\", getName(), repairFile.getName());\n TombstoneFileIterator iterator = newIteratorWithCheckForDataCorruption();\n int count = 0;\n while (iterator.hasNext()) {\n TombstoneEntry entry = iterator.next();\n if (entry == null) {\n logger.info(\"Found a corrupted entry in tombstone file {} after copying {} entries.\", getName(), count);\n break;\n }\n count++;\n repairFile.write(entry);\n }\n logger.info(\"Recovered {} records from file {} with size {}. Size after repair {}.\", count, getName(), getSize(), repairFile.getSize());\n repairFile.flushToDisk();\n Files.move(repairFile.getPath(), getPath(), REPLACE_EXISTING, ATOMIC_MOVE);\n dbDirectory.syncMetaData(); \n repairFile.close();\n close();\n open();\n return this;\n }\n\n private TombstoneFile createRepairFile() throws IOException {\n File repairFile = dbDirectory.getPath().resolve(getName()+\".repair\").toFile();\n while (!repairFile.createNewFile()) {\n logger.info(\"Repair file {} already exists, probably from a previous repair which failed. Deleting a trying again\", repairFile.getName());\n repairFile.delete();\n }\n\n TombstoneFile tombstoneFile = new TombstoneFile(repairFile, options, dbDirectory);\n tombstoneFile.open();\n return tombstoneFile;\n }\n\n String getName() {\n return backingFile.getName();\n }\n\n private Path getPath() {\n return backingFile.toPath();\n }\n\n private long getSize() {\n return backingFile.length();\n }\n\n TombstoneFile.TombstoneFileIterator newIterator() throws IOException {\n return new TombstoneFile.TombstoneFileIterator(false);\n }\n\n // Returns null when it finds a corrupted entry.\n TombstoneFile.TombstoneFileIterator newIteratorWithCheckForDataCorruption() throws IOException {\n return new TombstoneFile.TombstoneFileIterator(true);\n }\n\n private static File getTombstoneFile(DBDirectory dbDirectory, int fileId) {\n return dbDirectory.getPath().resolve(fileId + TOMBSTONE_FILE_NAME).toFile();\n }\n\n class TombstoneFileIterator implements Iterator {\n\n private final ByteBuffer buffer;\n private final boolean discardCorruptedRecords;\n\n TombstoneFileIterator(boolean discardCorruptedRecords) throws IOException {\n buffer = channel.map(FileChannel.MapMode.READ_ONLY, 0, channel.size());\n this.discardCorruptedRecords = discardCorruptedRecords;\n }\n\n @Override\n public boolean hasNext() {\n return buffer.hasRemaining();\n }\n\n @Override\n public TombstoneEntry next() {\n if (hasNext()) {\n if (discardCorruptedRecords)\n return TombstoneEntry.deserializeIfNotCorrupted(buffer);\n \n return TombstoneEntry.deserialize(buffer);\n }\n\n return null;\n }\n }\n}\n" }, { "path": "src/main/java/com/oath/halodb/Uns.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\n// This code is a derivative work heavily modified from the OHC project. See NOTICE file for copyright and license.\n\npackage com.oath.halodb;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport sun.misc.Unsafe;\n\nimport java.io.IOException;\nimport java.lang.reflect.Field;\nimport java.nio.Buffer;\nimport java.nio.ByteBuffer;\nimport java.nio.ByteOrder;\nimport java.util.Map;\nimport java.util.StringTokenizer;\nimport java.util.concurrent.ConcurrentHashMap;\nimport java.util.concurrent.ConcurrentMap;\n\nfinal class Uns {\n\n private static final Logger LOGGER = LoggerFactory.getLogger(Uns.class);\n\n private static final Unsafe unsafe;\n private static final NativeMemoryAllocator allocator;\n\n private static final boolean __DEBUG_OFF_HEAP_MEMORY_ACCESS = Boolean.parseBoolean(System.getProperty(OffHeapHashTableBuilder.SYSTEM_PROPERTY_PREFIX + \"debugOffHeapAccess\", \"false\"));\n private static final String __ALLOCATOR = System.getProperty(OffHeapHashTableBuilder.SYSTEM_PROPERTY_PREFIX + \"allocator\");\n\n //\n // #ifdef __DEBUG_OFF_HEAP_MEMORY_ACCESS\n //\n private static final ConcurrentMap ohDebug = __DEBUG_OFF_HEAP_MEMORY_ACCESS ? new ConcurrentHashMap(16384) : null;\n private static final Map ohFreeDebug = __DEBUG_OFF_HEAP_MEMORY_ACCESS ? new ConcurrentHashMap(16384) : null;\n\n private static final class AllocInfo {\n\n final long size;\n final Throwable trace;\n\n AllocInfo(Long size, Throwable trace) {\n this.size = size;\n this.trace = trace;\n }\n }\n\n static void clearUnsDebugForTest() {\n if (__DEBUG_OFF_HEAP_MEMORY_ACCESS) {\n try {\n if (!ohDebug.isEmpty()) {\n for (Map.Entry addrSize : ohDebug.entrySet()) {\n System.err.printf(\" still allocated: address=%d, size=%d%n\", addrSize.getKey(), addrSize.getValue().size);\n addrSize.getValue().trace.printStackTrace();\n }\n throw new RuntimeException(\"Not all allocated memory has been freed!\");\n }\n } finally {\n ohDebug.clear();\n ohFreeDebug.clear();\n }\n }\n }\n\n private static void freed(long address) {\n if (__DEBUG_OFF_HEAP_MEMORY_ACCESS) {\n AllocInfo allocInfo = ohDebug.remove(address);\n if (allocInfo == null) {\n Throwable freedAt = ohFreeDebug.get(address);\n throw new IllegalStateException(\"Free of unallocated region \" + address, freedAt);\n }\n ohFreeDebug.put(address, new Exception(\"free backtrace - t=\" + System.nanoTime()));\n }\n }\n\n private static void allocated(long address, long bytes) {\n if (__DEBUG_OFF_HEAP_MEMORY_ACCESS) {\n AllocInfo allocatedLen =\n ohDebug.putIfAbsent(address, new AllocInfo(bytes, new Exception(\"Thread: \" + Thread.currentThread())));\n if (allocatedLen != null) {\n throw new Error(\"Oops - allocate() got duplicate address\");\n }\n ohFreeDebug.remove(address);\n }\n }\n\n private static void validate(long address, long offset, long len) {\n if (__DEBUG_OFF_HEAP_MEMORY_ACCESS) {\n if (address == 0L) {\n throw new NullPointerException();\n }\n AllocInfo allocInfo = ohDebug.get(address);\n if (allocInfo == null) {\n Throwable freedAt = ohFreeDebug.get(address);\n throw new IllegalStateException(\"Access to unallocated region \" + address + \" - t=\" + System.nanoTime(), freedAt);\n }\n if (offset < 0L) {\n throw new IllegalArgumentException(\"Negative offset\");\n }\n if (len < 0L) {\n throw new IllegalArgumentException(\"Negative length\");\n }\n if (offset + len > allocInfo.size) {\n throw new IllegalArgumentException(\"Access outside allocated region\");\n }\n }\n }\n //\n // #endif\n //\n\n private static final UnsExt ext;\n\n static {\n try {\n Field field = Unsafe.class.getDeclaredField(\"theUnsafe\");\n field.setAccessible(true);\n unsafe = (Unsafe) field.get(null);\n if (unsafe.addressSize() > 8) {\n throw new RuntimeException(\"Address size \" + unsafe.addressSize() + \" not supported yet (max 8 bytes)\");\n }\n\n String javaVersion = System.getProperty(\"java.version\");\n if (javaVersion.indexOf('-') != -1) {\n javaVersion = javaVersion.substring(0, javaVersion.indexOf('-'));\n }\n StringTokenizer st = new StringTokenizer(javaVersion, \".\");\n int major = Integer.parseInt(st.nextToken());\n int minor = st.hasMoreTokens() ? Integer.parseInt(st.nextToken()) : 0;\n UnsExt e;\n if (major > 1 || minor >= 8) {\n try {\n // use new Java8 methods in sun.misc.Unsafe\n ext = new UnsExt8(unsafe);\n LOGGER.info(\"OHC using Java8 Unsafe API\");\n } catch (VirtualMachineError ex) {\n throw ex;\n }\n } else {\n throw new RuntimeException(\"HaloDB requires java version >= 1.8\");\n }\n\n if (__DEBUG_OFF_HEAP_MEMORY_ACCESS)\n LOGGER.warn(\"Degraded performance due to off-heap memory allocations and access guarded by debug code enabled via system property \" + OffHeapHashTableBuilder.SYSTEM_PROPERTY_PREFIX + \"debugOffHeapAccess=true\");\n\n NativeMemoryAllocator alloc;\n String allocType = __ALLOCATOR != null ? __ALLOCATOR : \"jna\";\n switch (allocType) {\n case \"unsafe\":\n alloc = new UnsafeAllocator();\n LOGGER.info(\"OHC using sun.misc.Unsafe memory allocation\");\n break;\n case \"jna\":\n default:\n alloc = new JNANativeAllocator();\n LOGGER.info(\"OHC using JNA OS native malloc/free\");\n }\n\n allocator = alloc;\n } catch (Exception e) {\n throw new AssertionError(e);\n }\n }\n\n private Uns() {\n }\n\n static long getLongFromByteArray(byte[] array, int offset) {\n if (offset < 0 || offset + 8 > array.length)\n throw new ArrayIndexOutOfBoundsException();\n return unsafe.getLong(array, (long) Unsafe.ARRAY_BYTE_BASE_OFFSET + offset);\n }\n\n static int getIntFromByteArray(byte[] array, int offset) {\n if (offset < 0 || offset + 4 > array.length) {\n throw new ArrayIndexOutOfBoundsException();\n }\n return unsafe.getInt(array, (long) Unsafe.ARRAY_BYTE_BASE_OFFSET + offset);\n }\n\n static short getShortFromByteArray(byte[] array, int offset) {\n if (offset < 0 || offset + 2 > array.length) {\n throw new ArrayIndexOutOfBoundsException();\n }\n return unsafe.getShort(array, (long) Unsafe.ARRAY_BYTE_BASE_OFFSET + offset);\n }\n\n static long getAndPutLong(long address, long offset, long value) {\n validate(address, offset, 8L);\n\n return ext.getAndPutLong(address, offset, value);\n }\n\n static void putLong(long address, long offset, long value) {\n validate(address, offset, 8L);\n unsafe.putLong(null, address + offset, value);\n }\n\n static long getLong(long address, long offset) {\n validate(address, offset, 8L);\n return unsafe.getLong(null, address + offset);\n }\n\n static void putInt(long address, long offset, int value) {\n validate(address, offset, 4L);\n unsafe.putInt(null, address + offset, value);\n }\n\n static int getInt(long address, long offset) {\n validate(address, offset, 4L);\n return unsafe.getInt(null, address + offset);\n }\n\n static void putShort(long address, long offset, short value) {\n validate(address, offset, 2L);\n unsafe.putShort(null, address + offset, value);\n }\n\n static short getShort(long address, long offset) {\n validate(address, offset, 2L);\n return unsafe.getShort(null, address + offset);\n }\n\n static void putByte(long address, long offset, byte value) {\n validate(address, offset, 1L);\n unsafe.putByte(null, address + offset, value);\n }\n\n static byte getByte(long address, long offset) {\n validate(address, offset, 1L);\n return unsafe.getByte(null, address + offset);\n }\n\n static boolean decrement(long address, long offset) {\n validate(address, offset, 4L);\n long v = ext.getAndAddInt(address, offset, -1);\n return v == 1;\n }\n\n static void increment(long address, long offset) {\n validate(address, offset, 4L);\n ext.getAndAddInt(address, offset, 1);\n }\n\n static void copyMemory(byte[] arr, int off, long address, long offset, long len) {\n validate(address, offset, len);\n unsafe.copyMemory(arr, Unsafe.ARRAY_BYTE_BASE_OFFSET + off, null, address + offset, len);\n }\n\n static void copyMemory(long address, long offset, byte[] arr, int off, long len) {\n validate(address, offset, len);\n unsafe.copyMemory(null, address + offset, arr, Unsafe.ARRAY_BYTE_BASE_OFFSET + off, len);\n }\n\n static void copyMemory(long src, long srcOffset, long dst, long dstOffset, long len) {\n validate(src, srcOffset, len);\n validate(dst, dstOffset, len);\n unsafe.copyMemory(null, src + srcOffset, null, dst + dstOffset, len);\n }\n\n static void setMemory(long address, long offset, long len, byte val) {\n validate(address, offset, len);\n unsafe.setMemory(address + offset, len, val);\n }\n\n static boolean memoryCompare(long adr1, long off1, long adr2, long off2, long len) {\n if (adr1 == 0L) {\n return false;\n }\n\n if (adr1 == adr2) {\n assert off1 == off2;\n return true;\n }\n\n for (; len >= 8; len -= 8, off1 += 8, off2 += 8) {\n if (Uns.getLong(adr1, off1) != Uns.getLong(adr2, off2)) {\n return false;\n }\n }\n for (; len >= 4; len -= 4, off1 += 4, off2 += 4) {\n if (Uns.getInt(adr1, off1) != Uns.getInt(adr2, off2)) {\n return false;\n }\n }\n for (; len >= 2; len -= 2, off1 += 2, off2 += 2) {\n if (Uns.getShort(adr1, off1) != Uns.getShort(adr2, off2)) {\n return false;\n }\n }\n for (; len > 0; len--, off1++, off2++) {\n if (Uns.getByte(adr1, off1) != Uns.getByte(adr2, off2)) {\n return false;\n }\n }\n\n return true;\n }\n\n static long crc32(long address, long offset, long len) {\n validate(address, offset, len);\n return ext.crc32(address, offset, len);\n }\n\n static long getTotalAllocated() {\n return allocator.getTotalAllocated();\n }\n\n static long allocate(long bytes) {\n return allocate(bytes, false);\n }\n\n static long allocate(long bytes, boolean throwOOME) {\n long address = allocator.allocate(bytes);\n if (address != 0L) {\n allocated(address, bytes);\n } else if (throwOOME) {\n throw new OutOfMemoryError(\"unable to allocate \" + bytes + \" in off-heap\");\n }\n return address;\n }\n\n static long allocateIOException(long bytes) throws IOException {\n return allocateIOException(bytes, false);\n }\n\n static long allocateIOException(long bytes, boolean throwOOME) throws IOException {\n long address = allocate(bytes, throwOOME);\n if (address == 0L) {\n throw new IOException(\"unable to allocate \" + bytes + \" in off-heap\");\n }\n return address;\n }\n\n static void free(long address) {\n if (address == 0L) {\n return;\n }\n freed(address);\n allocator.free(address);\n }\n\n private static final Class DIRECT_BYTE_BUFFER_CLASS;\n private static final Class DIRECT_BYTE_BUFFER_CLASS_R;\n private static final long DIRECT_BYTE_BUFFER_ADDRESS_OFFSET;\n private static final long DIRECT_BYTE_BUFFER_CAPACITY_OFFSET;\n private static final long DIRECT_BYTE_BUFFER_LIMIT_OFFSET;\n\n static {\n try {\n ByteBuffer directBuffer = ByteBuffer.allocateDirect(0);\n ByteBuffer directReadOnly = directBuffer.asReadOnlyBuffer();\n Class clazz = directBuffer.getClass();\n Class clazzReadOnly = directReadOnly.getClass();\n DIRECT_BYTE_BUFFER_ADDRESS_OFFSET = unsafe.objectFieldOffset(Buffer.class.getDeclaredField(\"address\"));\n DIRECT_BYTE_BUFFER_CAPACITY_OFFSET = unsafe.objectFieldOffset(Buffer.class.getDeclaredField(\"capacity\"));\n DIRECT_BYTE_BUFFER_LIMIT_OFFSET = unsafe.objectFieldOffset(Buffer.class.getDeclaredField(\"limit\"));\n DIRECT_BYTE_BUFFER_CLASS = clazz;\n DIRECT_BYTE_BUFFER_CLASS_R = clazzReadOnly;\n } catch (NoSuchFieldException e) {\n throw new RuntimeException(e);\n }\n }\n\n static ByteBuffer directBufferFor(long address, long offset, long len, boolean readOnly) {\n if (len > Integer.MAX_VALUE || len < 0L) {\n throw new IllegalArgumentException();\n }\n try {\n ByteBuffer bb = (ByteBuffer) unsafe.allocateInstance(readOnly ? DIRECT_BYTE_BUFFER_CLASS_R : DIRECT_BYTE_BUFFER_CLASS);\n unsafe.putLong(bb, DIRECT_BYTE_BUFFER_ADDRESS_OFFSET, address + offset);\n unsafe.putInt(bb, DIRECT_BYTE_BUFFER_CAPACITY_OFFSET, (int) len);\n unsafe.putInt(bb, DIRECT_BYTE_BUFFER_LIMIT_OFFSET, (int) len);\n bb.order(ByteOrder.BIG_ENDIAN);\n return bb;\n } catch (Error e) {\n throw e;\n } catch (Throwable t) {\n throw new RuntimeException(t);\n }\n }\n\n static void invalidateDirectBuffer(ByteBuffer buffer) {\n buffer.position(0);\n unsafe.putInt(buffer, DIRECT_BYTE_BUFFER_CAPACITY_OFFSET, 0);\n unsafe.putInt(buffer, DIRECT_BYTE_BUFFER_LIMIT_OFFSET, 0);\n unsafe.putLong(buffer, DIRECT_BYTE_BUFFER_ADDRESS_OFFSET, 0L);\n }\n\n static ByteBuffer readOnlyBuffer(long hashEntryAdr, int length, long offset) {\n return Uns.directBufferFor(hashEntryAdr + offset, 0, length, true);\n }\n\n static ByteBuffer buffer(long hashEntryAdr, long length, long offset) {\n return Uns.directBufferFor(hashEntryAdr + offset, 0, length, false);\n }\n}\n" }, { "path": "src/main/java/com/oath/halodb/UnsExt.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\n// This code is a derivative work heavily modified from the OHC project. See NOTICE file for copyright and license.\n\npackage com.oath.halodb;\n\nimport sun.misc.Unsafe;\n\nabstract class UnsExt {\n\n final Unsafe unsafe;\n\n UnsExt(Unsafe unsafe) {\n this.unsafe = unsafe;\n }\n\n abstract long getAndPutLong(long address, long offset, long value);\n\n abstract int getAndAddInt(long address, long offset, int value);\n\n abstract long crc32(long address, long offset, long len);\n}\n" }, { "path": "src/main/java/com/oath/halodb/UnsExt8.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\n// This code is a derivative work heavily modified from the OHC project. See NOTICE file for copyright and license.\n\npackage com.oath.halodb;\n\nimport sun.misc.Unsafe;\n\nimport java.util.zip.CRC32;\n\nfinal class UnsExt8 extends UnsExt {\n\n UnsExt8(Unsafe unsafe) {\n super(unsafe);\n }\n\n long getAndPutLong(long address, long offset, long value) {\n return unsafe.getAndSetLong(null, address + offset, value);\n }\n\n int getAndAddInt(long address, long offset, int value) {\n return unsafe.getAndAddInt(null, address + offset, value);\n }\n\n long crc32(long address, long offset, long len) {\n CRC32 crc = new CRC32();\n crc.update(Uns.directBufferFor(address, offset, len, true));\n long h = crc.getValue();\n h |= h << 32;\n return h;\n }\n}\n" }, { "path": "src/main/java/com/oath/halodb/UnsafeAllocator.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\n// This code is a derivative work heavily modified from the OHC project. See NOTICE file for copyright and license.\n\npackage com.oath.halodb;\n\nimport sun.misc.Unsafe;\n\nimport java.lang.reflect.Field;\n\nfinal class UnsafeAllocator implements NativeMemoryAllocator {\n\n static final Unsafe unsafe;\n\n static {\n try {\n Field field = Unsafe.class.getDeclaredField(\"theUnsafe\");\n field.setAccessible(true);\n unsafe = (Unsafe) field.get(null);\n } catch (Exception e) {\n throw new AssertionError(e);\n }\n }\n\n public long allocate(long size) {\n try {\n return unsafe.allocateMemory(size);\n } catch (OutOfMemoryError oom) {\n return 0L;\n }\n }\n\n public void free(long peer) {\n unsafe.freeMemory(peer);\n }\n\n public long getTotalAllocated() {\n return -1L;\n }\n}\n" }, { "path": "src/main/java/com/oath/halodb/Utils.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nclass Utils {\n static long roundUpToPowerOf2(long number) {\n return (number > 1) ? Long.highestOneBit((number - 1) << 1) : 1;\n }\n\n static int getValueOffset(int recordOffset, byte[] key) {\n return recordOffset + Record.Header.HEADER_SIZE + key.length;\n }\n\n //TODO: probably belongs to Record.\n static int getRecordSize(int keySize, int valueSize) {\n return keySize + valueSize + Record.Header.HEADER_SIZE;\n }\n\n static int getValueSize(int recordSize, byte[] key) {\n return recordSize - Record.Header.HEADER_SIZE - key.length;\n }\n\n static InMemoryIndexMetaData getMetaData(IndexFileEntry entry, int fileId) {\n return new InMemoryIndexMetaData(fileId, Utils.getValueOffset(entry.getRecordOffset(), entry.getKey()), Utils.getValueSize(entry.getRecordSize(), entry.getKey()), entry.getSequenceNumber());\n }\n\n static long toUnsignedIntFromInt(int value) {\n return value & 0xffffffffL;\n }\n\n static int toSignedIntFromLong(long value) {\n return (int)(value & 0xffffffffL);\n }\n\n static int toUnsignedByte(byte value) {\n return value & 0xFF;\n }\n}\n" }, { "path": "src/main/java/com/oath/halodb/Versions.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nclass Versions {\n\n static final int CURRENT_DATA_FILE_VERSION = 0;\n static final int CURRENT_INDEX_FILE_VERSION = 0;\n static final int CURRENT_TOMBSTONE_FILE_VERSION = 0;\n static final int CURRENT_META_FILE_VERSION = 0;\n}\n" }, { "path": "src/main/java/com/oath/halodb/histo/EstimatedHistogram.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\n// This code is a derivative work heavily modified from the OHC project. See NOTICE file for copyright and license.\n\npackage com.oath.halodb.histo;\n\nimport com.google.common.base.Objects;\n\nimport org.slf4j.Logger;\n\nimport java.util.Arrays;\nimport java.util.concurrent.atomic.AtomicLongArray;\n\npublic class EstimatedHistogram {\n\n /**\n * The series of values to which the counts in `buckets` correspond: 1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 14, 17, 20,\n * etc. Thus, a `buckets` of [0, 0, 1, 10] would mean we had seen one value of 3 and 10 values of 4.\n *\n * The series starts at 1 and grows by 1.2 each time (rounding and removing duplicates). It goes from 1 to around\n * 36M by default (creating 90+1 buckets), which will give us timing resolution from microseconds to 36 seconds,\n * with less precision as the numbers get larger.\n *\n * Each bucket represents values from (previous bucket offset, current offset].\n */\n private final long[] bucketOffsets;\n\n // buckets is one element longer than bucketOffsets -- the last element is values greater than the last offset\n final AtomicLongArray buckets;\n\n public EstimatedHistogram() {\n this(90);\n }\n\n public EstimatedHistogram(int bucketCount) {\n bucketOffsets = newOffsets(bucketCount);\n buckets = new AtomicLongArray(bucketOffsets.length + 1);\n }\n\n public EstimatedHistogram(long[] offsets, long[] bucketData) {\n assert bucketData.length == offsets.length + 1;\n bucketOffsets = offsets;\n buckets = new AtomicLongArray(bucketData);\n }\n\n private static long[] newOffsets(int size) {\n long[] result = new long[size];\n long last = 1;\n result[0] = last;\n for (int i = 1; i < size; i++) {\n long next = Math.round(last * 1.2);\n if (next == last) {\n next++;\n }\n result[i] = next;\n last = next;\n }\n\n return result;\n }\n\n /**\n * @return the histogram values corresponding to each bucket index\n */\n public long[] getBucketOffsets() {\n return bucketOffsets;\n }\n\n /**\n * Increments the count of the bucket closest to n, rounding UP.\n */\n public void add(long n) {\n int index = Arrays.binarySearch(bucketOffsets, n);\n if (index < 0) {\n // inexact match, take the first bucket higher than n\n index = -index - 1;\n }\n // else exact match; we're good\n buckets.incrementAndGet(index);\n }\n\n /**\n * @return the count in the given bucket\n */\n long get(int bucket) {\n return buckets.get(bucket);\n }\n\n /**\n * @param reset zero out buckets afterwards if true\n * @return a long[] containing the current histogram buckets\n */\n public long[] getBuckets(boolean reset) {\n final int len = buckets.length();\n long[] rv = new long[len];\n\n if (reset) {\n for (int i = 0; i < len; i++) {\n rv[i] = buckets.getAndSet(i, 0L);\n }\n } else {\n for (int i = 0; i < len; i++) {\n rv[i] = buckets.get(i);\n }\n }\n\n return rv;\n }\n\n /**\n * @return the smallest value that could have been added to this histogram\n */\n public long min() {\n for (int i = 0; i < buckets.length(); i++) {\n if (buckets.get(i) > 0) {\n return i == 0 ? 0 : 1 + bucketOffsets[i - 1];\n }\n }\n return 0;\n }\n\n /**\n * @return the largest value that could have been added to this histogram. If the histogram overflowed, returns\n * Long.MAX_VALUE.\n */\n public long max() {\n int lastBucket = buckets.length() - 1;\n if (buckets.get(lastBucket) > 0) {\n return Long.MAX_VALUE;\n }\n\n for (int i = lastBucket - 1; i >= 0; i--) {\n if (buckets.get(i) > 0) {\n return bucketOffsets[i];\n }\n }\n return 0;\n }\n\n /**\n * @return estimated value at given percentile\n */\n public long percentile(double percentile) {\n assert percentile >= 0 && percentile <= 1.0;\n int lastBucket = buckets.length() - 1;\n if (buckets.get(lastBucket) > 0) {\n throw new IllegalStateException(\"Unable to compute when histogram overflowed\");\n }\n\n long pcount = (long) Math.floor(count() * percentile);\n if (pcount == 0) {\n return 0;\n }\n\n long elements = 0;\n for (int i = 0; i < lastBucket; i++) {\n elements += buckets.get(i);\n if (elements >= pcount) {\n return bucketOffsets[i];\n }\n }\n return 0;\n }\n\n /**\n * @return the mean histogram value (average of bucket offsets, weighted by count)\n * @throws IllegalStateException if any values were greater than the largest bucket threshold\n */\n public long mean() {\n int lastBucket = buckets.length() - 1;\n if (buckets.get(lastBucket) > 0) {\n throw new IllegalStateException(\"Unable to compute ceiling for max when histogram overflowed\");\n }\n\n long elements = 0;\n long sum = 0;\n for (int i = 0; i < lastBucket; i++) {\n long bCount = buckets.get(i);\n elements += bCount;\n sum += bCount * bucketOffsets[i];\n }\n\n return (long) Math.ceil((double) sum / elements);\n }\n\n /**\n * @return the total number of non-zero values\n */\n public long count() {\n long sum = 0L;\n for (int i = 0; i < buckets.length(); i++) {\n sum += buckets.get(i);\n }\n return sum;\n }\n\n /**\n * @return true if this histogram has overflowed -- that is, a value larger than our largest bucket could bound was\n * added\n */\n public boolean isOverflowed() {\n return buckets.get(buckets.length() - 1) > 0;\n }\n\n /**\n * log.debug() every record in the histogram\n */\n public void log(Logger log) {\n // only print overflow if there is any\n int nameCount;\n if (buckets.get(buckets.length() - 1) == 0) {\n nameCount = buckets.length() - 1;\n } else {\n nameCount = buckets.length();\n }\n String[] names = new String[nameCount];\n\n int maxNameLength = 0;\n for (int i = 0; i < nameCount; i++) {\n names[i] = nameOfRange(bucketOffsets, i);\n maxNameLength = Math.max(maxNameLength, names[i].length());\n }\n\n // emit log records\n String formatstr = \"%\" + maxNameLength + \"s: %d\";\n for (int i = 0; i < nameCount; i++) {\n long count = buckets.get(i);\n // sort-of-hack to not print empty ranges at the start that are only used to demarcate the\n // first populated range. for code clarity we don't omit this record from the maxNameLength\n // calculation, and accept the unnecessary whitespace prefixes that will occasionally occur\n if (i == 0 && count == 0) {\n continue;\n }\n log.debug(String.format(formatstr, names[i], count));\n }\n }\n\n public String toString() {\n // only print overflow if there is any\n int nameCount;\n if (buckets.get(buckets.length() - 1) == 0) {\n nameCount = buckets.length() - 1;\n } else {\n nameCount = buckets.length();\n }\n String[] names = new String[nameCount];\n\n int maxNameLength = 0;\n for (int i = 0; i < nameCount; i++) {\n names[i] = nameOfRange(bucketOffsets, i);\n maxNameLength = Math.max(maxNameLength, names[i].length());\n }\n\n StringBuilder sb = new StringBuilder();\n\n // emit log records\n String formatstr = \"%\" + maxNameLength + \"s: %d\\n\";\n for (int i = 0; i < nameCount; i++) {\n long count = buckets.get(i);\n // sort-of-hack to not print empty ranges at the start that are only used to demarcate the\n // first populated range. for code clarity we don't omit this record from the maxNameLength\n // calculation, and accept the unnecessary whitespace prefixes that will occasionally occur\n if (i == 0 && count == 0) {\n continue;\n }\n sb.append(String.format(formatstr, names[i], count));\n }\n\n return sb.toString();\n }\n\n private static String nameOfRange(long[] bucketOffsets, int index) {\n StringBuilder sb = new StringBuilder();\n appendRange(sb, bucketOffsets, index);\n return sb.toString();\n }\n\n private static void appendRange(StringBuilder sb, long[] bucketOffsets, int index) {\n sb.append(\"[\");\n if (index == 0) {\n if (bucketOffsets[0] > 0)\n // by original definition, this histogram is for values greater than zero only;\n // if values of 0 or less are required, an entry of lb-1 must be inserted at the start\n {\n sb.append(\"1\");\n } else {\n sb.append(\"-Inf\");\n }\n } else {\n sb.append(bucketOffsets[index - 1] + 1);\n }\n sb.append(\"..\");\n if (index == bucketOffsets.length) {\n sb.append(\"Inf\");\n } else {\n sb.append(bucketOffsets[index]);\n }\n sb.append(\"]\");\n }\n\n @Override\n public boolean equals(Object o) {\n if (this == o) {\n return true;\n }\n\n if (!(o instanceof EstimatedHistogram)) {\n return false;\n }\n\n EstimatedHistogram that = (EstimatedHistogram) o;\n return Arrays.equals(getBucketOffsets(), that.getBucketOffsets()) &&\n Arrays.equals(getBuckets(false), that.getBuckets(false));\n }\n\n @Override\n public int hashCode() {\n return Objects.hashCode(getBucketOffsets(), getBuckets(false));\n }\n}\n" }, { "path": "src/test/java/com/oath/halodb/CheckOffHeapHashTable.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\n// This code is a derivative work heavily modified from the OHC project. See NOTICE file for copyright and license.\n\npackage com.oath.halodb;\n\nimport com.oath.halodb.histo.EstimatedHistogram;\n\nimport java.nio.ByteBuffer;\nimport java.util.concurrent.atomic.AtomicLong;\n\n/**\n * This is a {@link OffHeapHashTable} implementation used to validate functionality of\n * {@link OffHeapHashTableImpl} - this implementation is not for production use!\n */\nfinal class CheckOffHeapHashTable implements OffHeapHashTable\n{\n private final HashTableValueSerializer valueSerializer;\n\n private final CheckSegment[] maps;\n private final int segmentShift;\n private final long segmentMask;\n private final float loadFactor;\n private long putFailCount;\n private final Hasher hasher;\n\n CheckOffHeapHashTable(OffHeapHashTableBuilder builder)\n {\n loadFactor = builder.getLoadFactor();\n hasher = Hasher.create(builder.getHashAlgorighm());\n\n int segments = builder.getSegmentCount();\n int bitNum = HashTableUtil.bitNum(segments) - 1;\n this.segmentShift = 64 - bitNum;\n this.segmentMask = ((long) segments - 1) << segmentShift;\n\n maps = new CheckSegment[segments];\n for (int i = 0; i < maps.length; i++)\n maps[i] = new CheckSegment(builder.getHashTableSize(), builder.getLoadFactor());\n\n valueSerializer = builder.getValueSerializer();\n }\n\n public boolean put(byte[] key, V value)\n {\n KeyBuffer keyBuffer = keySource(key);\n byte[] data = value(value);\n\n CheckSegment segment = segment(keyBuffer.hash());\n return segment.put(keyBuffer, data, false, null);\n }\n\n public boolean addOrReplace(byte[] key, V old, V value)\n {\n KeyBuffer keyBuffer = keySource(key);\n byte[] data = value(value);\n byte[] oldData = value(old);\n\n CheckSegment segment = segment(keyBuffer.hash());\n return segment.put(keyBuffer, data, false, oldData);\n }\n\n public boolean putIfAbsent(byte[] key, V v)\n {\n KeyBuffer keyBuffer = keySource(key);\n byte[] data = value(v);\n\n CheckSegment segment = segment(keyBuffer.hash());\n return segment.put(keyBuffer, data, true, null);\n }\n\n public boolean putIfAbsent(byte[] key, V value, long expireAt)\n {\n throw new UnsupportedOperationException();\n }\n\n public boolean put(byte[] key, V value, long expireAt)\n {\n throw new UnsupportedOperationException();\n }\n\n public boolean remove(byte[] key)\n {\n KeyBuffer keyBuffer = keySource(key);\n CheckSegment segment = segment(keyBuffer.hash());\n return segment.remove(keyBuffer);\n }\n\n public void clear()\n {\n for (CheckSegment map : maps)\n map.clear();\n }\n\n public V get(byte[] key)\n {\n KeyBuffer keyBuffer = keySource(key);\n CheckSegment segment = segment(keyBuffer.hash());\n byte[] value = segment.get(keyBuffer);\n\n if (value == null)\n return null;\n\n return valueSerializer.deserialize(ByteBuffer.wrap(value));\n }\n\n public boolean containsKey(byte[] key)\n {\n KeyBuffer keyBuffer = keySource(key);\n CheckSegment segment = segment(keyBuffer.hash());\n return segment.get(keyBuffer) != null;\n }\n\n public void resetStatistics()\n {\n for (CheckSegment map : maps)\n map.resetStatistics();\n putFailCount = 0;\n }\n\n public long size()\n {\n long r = 0;\n for (CheckSegment map : maps)\n r += map.size();\n return r;\n }\n\n public int[] hashTableSizes()\n {\n // no hash table size info\n return new int[maps.length];\n }\n\n public SegmentStats[] perSegmentStats() {\n SegmentStats[] stats = new SegmentStats[maps.length];\n for (int i = 0; i < stats.length; i++) {\n CheckSegment map = maps[i];\n stats[i] = new SegmentStats(map.size(), -1, -1, -1);\n }\n\n return stats;\n }\n\n public EstimatedHistogram getBucketHistogram()\n {\n throw new UnsupportedOperationException();\n }\n\n public int segments()\n {\n return maps.length;\n }\n\n public float loadFactor()\n {\n return loadFactor;\n }\n\n public OffHeapHashTableStats stats()\n {\n return new OffHeapHashTableStats(\n hitCount(),\n missCount(),\n size(),\n -1L,\n putAddCount(),\n putReplaceCount(),\n putFailCount,\n removeCount(),\n perSegmentStats()\n );\n }\n\n private long putAddCount()\n {\n long putAddCount = 0L;\n for (CheckSegment map : maps)\n putAddCount += map.putAddCount;\n return putAddCount;\n }\n\n private long putReplaceCount()\n {\n long putReplaceCount = 0L;\n for (CheckSegment map : maps)\n putReplaceCount += map.putReplaceCount;\n return putReplaceCount;\n }\n\n private long removeCount()\n {\n long removeCount = 0L;\n for (CheckSegment map : maps)\n removeCount += map.removeCount;\n return removeCount;\n }\n\n private long hitCount()\n {\n long hitCount = 0L;\n for (CheckSegment map : maps)\n hitCount += map.hitCount;\n return hitCount;\n }\n\n private long missCount()\n {\n long missCount = 0L;\n for (CheckSegment map : maps)\n missCount += map.missCount;\n return missCount;\n }\n\n public void close()\n {\n clear();\n }\n\n //\n //\n //\n\n private CheckSegment segment(long hash)\n {\n int seg = (int) ((hash & segmentMask) >>> segmentShift);\n return maps[seg];\n }\n\n KeyBuffer keySource(byte[] key) {\n KeyBuffer keyBuffer = new KeyBuffer(key);\n return keyBuffer.finish(hasher);\n }\n\n private byte[] value(V value)\n {\n if (value == null) {\n return null;\n }\n ByteBuffer buf = ByteBuffer.allocate(valueSerializer.serializedSize(value));\n valueSerializer.serialize(value, buf);\n return buf.array();\n }\n}\n" }, { "path": "src/test/java/com/oath/halodb/CheckSegment.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\n// This code is a derivative work heavily modified from the OHC project. See NOTICE file for copyright and license.\n\npackage com.oath.halodb;\n\nimport java.util.Arrays;\nimport java.util.HashMap;\nimport java.util.LinkedList;\nimport java.util.Map;\nimport java.util.concurrent.atomic.AtomicLong;\n\n/**\n * On-heap test-only counterpart of {@link SegmentNonMemoryPool} for {@link CheckOffHeapHashTable}.\n */\nfinal class CheckSegment {\n private final Map map;\n private final LinkedList lru = new LinkedList<>();\n\n long hitCount;\n long missCount;\n long putAddCount;\n long putReplaceCount;\n long removeCount;\n long evictedEntries;\n\n public CheckSegment(int initialCapacity, float loadFactor) {\n this.map = new HashMap<>(initialCapacity, loadFactor);\n }\n\n synchronized void clear()\n {\n map.clear();\n lru.clear();\n }\n\n synchronized byte[] get(KeyBuffer keyBuffer)\n {\n byte[] r = map.get(keyBuffer);\n if (r == null)\n {\n missCount++;\n return null;\n }\n\n lru.remove(keyBuffer);\n lru.addFirst(keyBuffer);\n hitCount++;\n\n return r;\n }\n\n synchronized boolean put(KeyBuffer keyBuffer, byte[] data, boolean ifAbsent, byte[] old)\n {\n byte[] existing = map.get(keyBuffer);\n\n if (ifAbsent && existing != null)\n return false;\n\n if (old != null && !Arrays.equals(old, existing)) {\n return false;\n }\n\n map.put(keyBuffer, data);\n lru.remove(keyBuffer);\n lru.addFirst(keyBuffer);\n\n if (existing != null)\n {\n putReplaceCount++;\n }\n else\n putAddCount++;\n\n return true;\n }\n\n synchronized boolean remove(KeyBuffer keyBuffer)\n {\n byte[] old = map.remove(keyBuffer);\n if (old != null)\n {\n boolean r = lru.remove(keyBuffer);\n removeCount++;\n return r;\n }\n return false;\n }\n\n synchronized long size()\n {\n return map.size();\n }\n\n static long sizeOf(KeyBuffer key, byte[] value)\n {\n // calculate the same value as the original impl would do\n return NonMemoryPoolHashEntries.ENTRY_OFF_DATA + key.size() + value.length;\n }\n\n void resetStatistics()\n {\n evictedEntries = 0L;\n hitCount = 0L;\n missCount = 0L;\n putAddCount = 0L;\n putReplaceCount = 0L;\n removeCount = 0L;\n }\n\n}\n" }, { "path": "src/test/java/com/oath/halodb/CompactionWithErrorsTest.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport com.google.common.util.concurrent.RateLimiter;\n\nimport org.testng.Assert;\nimport org.testng.annotations.Test;\n\nimport sun.nio.ch.FileChannelImpl;\n\nimport java.io.IOException;\nimport java.nio.channels.WritableByteChannel;\nimport java.nio.file.Paths;\nimport java.util.List;\n\nimport mockit.Expectations;\nimport mockit.Invocation;\nimport mockit.Mock;\nimport mockit.MockUp;\nimport mockit.Mocked;\nimport mockit.VerificationsInOrder;\n\npublic class CompactionWithErrorsTest extends TestBase {\n\n @Test\n public void testCompactionWithException() throws HaloDBException, InterruptedException {\n\n new MockUp() {\n private int callCount = 0;\n\n @Mock\n public double acquire(int permits) {\n if (++callCount == 3) {\n // throw an exception when copying the third record. \n throw new OutOfMemoryError(\"Throwing mock exception form compaction thread.\");\n }\n return 10;\n }\n };\n\n String directory = TestUtils.getTestDirectory(\"CompactionManagerTest\", \"testCompactionWithException\");\n\n HaloDBOptions options = new HaloDBOptions();\n options.setMaxFileSize(10 * 1024);\n options.setCompactionThresholdPerFile(0.5);\n\n HaloDB db = getTestDB(directory, options);\n int numberOfRecords = 30; // three files.\n\n List records = insertAndUpdate(db, numberOfRecords);\n\n TestUtils.waitForCompactionToComplete(db);\n\n // An exception was thrown while copying a record in the compaction thread.\n // Make sure that all records are still correct. \n Assert.assertEquals(db.size(), records.size());\n for (Record r : records) {\n Assert.assertEquals(db.get(r.getKey()), r.getValue());\n }\n\n db.close();\n db = getTestDBWithoutDeletingFiles(directory, options);\n\n // Make sure that everything is good after\n // we open the db again. Since compaction had failed\n // there would be two copies of the same record in two different files. \n Assert.assertEquals(db.size(), records.size());\n for (Record r : records) {\n Assert.assertEquals(db.get(r.getKey()), r.getValue());\n }\n }\n\n @Test\n public void testRestartCompactionThreadAfterCrash(@Mocked CompactionManager compactionManager) throws HaloDBException, InterruptedException, IOException {\n\n new Expectations(CompactionManager.class) {{\n // nothing mocked. call the real implementation.\n // this is used only for verifications later.\n }};\n\n new MockUp() {\n private int callCount = 0;\n\n @Mock\n public double acquire(int permits) {\n if (++callCount == 3 || callCount == 8) {\n // throw exceptions twice, each time compaction thread should crash and restart. \n throw new OutOfMemoryError(\"Throwing mock exception from compaction thread.\");\n }\n return 10;\n }\n };\n\n String directory = TestUtils.getTestDirectory(\"CompactionManagerTest\", \"testRestartCompactionThreadAfterCrash\");\n\n HaloDBOptions options = new HaloDBOptions();\n options.setMaxFileSize(10 * 1024);\n options.setCompactionThresholdPerFile(0.5);\n\n HaloDB db = getTestDB(directory, options);\n int numberOfRecords = 30; // three files, 10 record in each.\n\n List records = insertAndUpdate(db, numberOfRecords);\n\n TestUtils.waitForCompactionToComplete(db);\n\n // An exception was thrown while copying a record in the compaction thread.\n // Make sure that all records are still correct.\n Assert.assertEquals(db.size(), records.size());\n for (Record r : records) {\n Assert.assertEquals(db.get(r.getKey()), r.getValue());\n }\n\n db.close();\n db = getTestDBWithoutDeletingFiles(directory, options);\n\n // Make sure that everything is good after\n // we open the db again. Since compaction had failed\n // there would be two copies of the same record in two different files.\n Assert.assertEquals(db.size(), records.size());\n for (Record r : records) {\n Assert.assertEquals(db.get(r.getKey()), r.getValue());\n }\n\n new VerificationsInOrder() {{\n // called when db.open()\n compactionManager.startCompactionThread();\n\n // compaction thread should have crashed twice and each time it should have been restarted. \n compactionManager.startCompactionThread();\n compactionManager.startCompactionThread();\n\n // called after db.close()\n compactionManager.stopCompactionThread(true);\n\n // called when db.open() the second time. \n compactionManager.startCompactionThread();\n }};\n\n DBMetaData dbMetaData = new DBMetaData(dbDirectory);\n dbMetaData.loadFromFileIfExists();\n\n // Since compaction thread was restarted after it crashed IOError flag must not be set.\n Assert.assertFalse(dbMetaData.isIOError());\n }\n\n @Test\n public void testCompactionThreadStopWithIOException() throws HaloDBException, InterruptedException, IOException {\n // Throw an IOException while stopping compaction thread.\n new MockUp() {\n\n @Mock\n boolean stopCompactionThread(boolean flag) throws IOException {\n throw new IOException(\"Throwing mock IOException while stopping compaction thread.\");\n\n }\n };\n\n String directory = TestUtils.getTestDirectory(\"CompactionManagerTest\", \"testCompactionThreadStopWithIOException\");\n\n HaloDBOptions options = new HaloDBOptions();\n options.setMaxFileSize(10 * 1024);\n options.setCompactionThresholdPerFile(0.5);\n\n HaloDB db = getTestDB(directory, options);\n int numberOfRecords = 20; // three files.\n\n insertAndUpdate(db, numberOfRecords);\n TestUtils.waitForCompactionToComplete(db);\n db.close();\n\n DBMetaData dbMetaData = new DBMetaData(dbDirectory);\n dbMetaData.loadFromFileIfExists();\n\n // Since there was an IOException while stopping compaction IOError flag must have been set. \n Assert.assertTrue(dbMetaData.isIOError());\n }\n\n private List insertAndUpdate(HaloDB db, int numberOfRecords) throws HaloDBException {\n List records = TestUtils.insertRandomRecordsOfSize(db, numberOfRecords, 1024 - Record.Header.HEADER_SIZE);\n\n // Update first 5 records in each file.\n for (int i = 0; i < 5; i++) {\n byte[] value = TestUtils.generateRandomByteArray();\n db.put(records.get(i).getKey(), value);\n records.set(i, new Record(records.get(i).getKey(), value));\n db.put(records.get(i+10).getKey(), value);\n records.set(i+10, new Record(records.get(i+10).getKey(), value));\n }\n return records;\n }\n}\n" }, { "path": "src/test/java/com/oath/halodb/CrossCheckTest.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\n// This code is a derivative work heavily modified from the OHC project. See NOTICE file for copyright and license.\n\npackage com.oath.halodb;\n\nimport com.google.common.primitives.Longs;\nimport com.oath.halodb.histo.EstimatedHistogram;\n\nimport org.testng.Assert;\nimport org.testng.annotations.AfterMethod;\nimport org.testng.annotations.DataProvider;\nimport org.testng.annotations.Test;\n\nimport java.io.IOException;\nimport java.util.ArrayList;\nimport java.util.HashMap;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.Random;\n\nimport static org.testng.Assert.assertEquals;\nimport static org.testng.Assert.assertTrue;\n\n// This unit test uses the production cache implementation and an independent OHCache implementation used to\n// cross-check the production implementation.\npublic class CrossCheckTest\n{\n\n private static final int fixedValueSize = 20;\n private static final int fixedKeySize = 16;\n\n @AfterMethod(alwaysRun = true)\n public void deinit()\n {\n Uns.clearUnsDebugForTest();\n }\n\n static DoubleCheckOffHeapHashTableImpl cache(HashAlgorithm hashAlgorithm, boolean useMemoryPool)\n {\n return cache(hashAlgorithm, useMemoryPool, 256);\n }\n\n static DoubleCheckOffHeapHashTableImpl cache(HashAlgorithm hashAlgorithm, boolean useMemoryPool, long capacity)\n {\n return cache(hashAlgorithm, useMemoryPool, capacity, -1);\n }\n\n static DoubleCheckOffHeapHashTableImpl cache(HashAlgorithm hashAlgorithm, boolean useMemoryPool, long capacity, int hashTableSize)\n {\n return cache(hashAlgorithm, useMemoryPool, capacity, hashTableSize, -1, -1);\n }\n\n static DoubleCheckOffHeapHashTableImpl cache(HashAlgorithm hashAlgorithm, boolean useMemoryPool, long capacity, int hashTableSize, int segments, long maxEntrySize)\n {\n OffHeapHashTableBuilder builder = OffHeapHashTableBuilder.newBuilder()\n .valueSerializer(HashTableTestUtils.byteArraySerializer)\n .hashMode(hashAlgorithm)\n .fixedValueSize(fixedValueSize);\n if (useMemoryPool)\n builder.useMemoryPool(true).fixedKeySize(fixedKeySize);\n\n if (hashTableSize > 0)\n builder.hashTableSize(hashTableSize);\n if (segments > 0)\n builder.segmentCount(segments);\n else\n // use 16 segments by default to prevent differing test behaviour on varying test hardware\n builder.segmentCount(16);\n\n return new DoubleCheckOffHeapHashTableImpl<>(builder);\n }\n\n @DataProvider(name = \"hashAlgorithms\")\n public Object[][] cacheEviction()\n {\n return new Object[][]{\n {HashAlgorithm.MURMUR3, false },\n {HashAlgorithm.MURMUR3, true },\n {HashAlgorithm.CRC32, false },\n {HashAlgorithm.CRC32, true },\n {HashAlgorithm.XX, false },\n {HashAlgorithm.XX, true }\n };\n }\n\n\n @Test(dataProvider = \"hashAlgorithms\")\n public void testBasics(HashAlgorithm hashAlgorithm, boolean useMemoryPool) throws IOException, InterruptedException\n {\n try (OffHeapHashTable cache = cache(hashAlgorithm, useMemoryPool))\n {\n byte[] key = HashTableTestUtils.randomBytes(12);\n byte[] value = HashTableTestUtils.randomBytes(fixedValueSize);\n cache.put(key, value);\n\n byte[] actual = cache.get(key);\n Assert.assertEquals(actual, value);\n\n cache.remove(key);\n\n Map keyValues = new HashMap<>();\n\n for (int i = 0; i < 100; i++) {\n byte[] k = HashTableTestUtils.randomBytes(8);\n byte[] v = HashTableTestUtils.randomBytes(fixedValueSize);\n keyValues.put(k, v);\n cache.put(k, v);\n }\n\n keyValues.forEach((k, v) -> {\n Assert.assertEquals(cache.get(k), v);\n });\n\n // implicitly compares stats\n cache.stats();\n }\n }\n\n @Test(dataProvider = \"hashAlgorithms\", dependsOnMethods = \"testBasics\")\n public void testManyValues(HashAlgorithm hashAlgorithm, boolean useMemoryPool) throws IOException, InterruptedException\n {\n try (OffHeapHashTable cache = cache(hashAlgorithm, useMemoryPool, 64, -1))\n {\n List entries = HashTableTestUtils.fillMany(cache, fixedValueSize);\n\n OffHeapHashTableStats stats = cache.stats();\n Assert.assertEquals(stats.getPutAddCount(), HashTableTestUtils.manyCount);\n Assert.assertEquals(stats.getSize(), HashTableTestUtils.manyCount);\n\n entries.forEach(kv -> Assert.assertEquals(cache.get(kv.key), kv.value));\n\n stats = cache.stats();\n Assert.assertEquals(stats.getHitCount(), HashTableTestUtils.manyCount);\n Assert.assertEquals(stats.getSize(), HashTableTestUtils.manyCount);\n\n for (int i = 0; i < HashTableTestUtils.manyCount; i++)\n {\n HashTableTestUtils.KeyValuePair kv = entries.get(i);\n Assert.assertEquals(cache.get(kv.key), kv.value, \"for i=\"+i);\n assertTrue(cache.containsKey(kv.key), \"for i=\"+i);\n byte[] updated = HashTableTestUtils.randomBytes(fixedValueSize);\n cache.put(kv.key, updated);\n entries.set(i, new HashTableTestUtils.KeyValuePair(kv.key, updated));\n Assert.assertEquals(cache.get(kv.key), updated, \"for i=\"+i);\n Assert.assertEquals(cache.size(), HashTableTestUtils.manyCount, \"for i=\" + i);\n assertTrue(cache.containsKey(kv.key), \"for i=\"+i);\n }\n\n stats = cache.stats();\n Assert.assertEquals(stats.getPutReplaceCount(), HashTableTestUtils.manyCount);\n Assert.assertEquals(stats.getSize(), HashTableTestUtils.manyCount);\n\n entries.forEach(kv -> Assert.assertEquals(cache.get(kv.key), kv.value));\n\n stats = cache.stats();\n Assert.assertEquals(stats.getHitCount(), HashTableTestUtils.manyCount * 6);\n Assert.assertEquals(stats.getSize(), HashTableTestUtils.manyCount);\n\n for (int i = 0; i < HashTableTestUtils.manyCount; i++)\n {\n HashTableTestUtils.KeyValuePair kv = entries.get(i);\n Assert.assertEquals(cache.get(kv.key), kv.value, \"for i=\" + i);\n assertTrue(cache.containsKey(kv.key), \"for i=\" + i);\n cache.remove(kv.key);\n Assert.assertNull(cache.get(kv.key), \"for i=\" + i);\n Assert.assertFalse(cache.containsKey(kv.key), \"for i=\" + i);\n Assert.assertEquals(cache.stats().getRemoveCount(), i + 1);\n Assert.assertEquals(cache.size(), HashTableTestUtils.manyCount - i - 1, \"for i=\" + i);\n }\n\n stats = cache.stats();\n Assert.assertEquals(stats.getRemoveCount(), HashTableTestUtils.manyCount);\n Assert.assertEquals(stats.getSize(), 0);\n }\n }\n\n\n @Test(dataProvider = \"hashAlgorithms\", dependsOnMethods = \"testBasics\")\n public void testRehash(HashAlgorithm hashAlgorithm, boolean useMemoryPool) throws IOException, InterruptedException\n {\n int count = 10_000;\n OffHeapHashTableBuilder builder = OffHeapHashTableBuilder.newBuilder()\n .valueSerializer(HashTableTestUtils.byteArraySerializer)\n .hashMode(hashAlgorithm)\n .fixedValueSize(fixedValueSize)\n .hashTableSize(count/4)\n .segmentCount(1)\n .loadFactor(1);\n\n if (useMemoryPool)\n builder.useMemoryPool(true).fixedKeySize(fixedKeySize);\n\n\n try (OffHeapHashTable cache = new DoubleCheckOffHeapHashTableImpl<>(builder))\n {\n List entries = HashTableTestUtils.fill(cache, fixedValueSize, count);\n\n OffHeapHashTableStats stats = cache.stats();\n Assert.assertEquals(stats.getPutAddCount(), count);\n Assert.assertEquals(stats.getSize(), count);\n Assert.assertEquals(stats.getRehashCount(), 2); // default load factor of 0.75, therefore 3 rehashes.\n\n entries.forEach(kv -> Assert.assertEquals(cache.get(kv.key), kv.value));\n\n stats = cache.stats();\n Assert.assertEquals(stats.getHitCount(), count);\n Assert.assertEquals(stats.getSize(), count);\n\n for (int i = 0; i < count; i++)\n {\n HashTableTestUtils.KeyValuePair kv = entries.get(i);\n Assert.assertEquals(cache.get(kv.key), kv.value, \"for i=\"+i);\n assertTrue(cache.containsKey(kv.key), \"for i=\"+i);\n byte[] updated = HashTableTestUtils.randomBytes(fixedValueSize);\n cache.put(kv.key, updated);\n entries.set(i, new HashTableTestUtils.KeyValuePair(kv.key, updated));\n Assert.assertEquals(cache.get(kv.key), updated, \"for i=\"+i);\n Assert.assertEquals(cache.size(), count, \"for i=\" + i);\n assertTrue(cache.containsKey(kv.key), \"for i=\"+i);\n }\n\n stats = cache.stats();\n Assert.assertEquals(stats.getPutReplaceCount(), count);\n Assert.assertEquals(stats.getSize(), count);\n Assert.assertEquals(stats.getRehashCount(), 2);\n\n entries.forEach(kv -> Assert.assertEquals(cache.get(kv.key), kv.value));\n\n stats = cache.stats();\n Assert.assertEquals(stats.getHitCount(), count * 6);\n Assert.assertEquals(stats.getSize(), count);\n\n for (int i = 0; i < count; i++)\n {\n HashTableTestUtils.KeyValuePair kv = entries.get(i);\n Assert.assertEquals(cache.get(kv.key), kv.value, \"for i=\" + i);\n assertTrue(cache.containsKey(kv.key), \"for i=\" + i);\n cache.remove(kv.key);\n Assert.assertNull(cache.get(kv.key), \"for i=\" + i);\n Assert.assertFalse(cache.containsKey(kv.key), \"for i=\" + i);\n Assert.assertEquals(cache.stats().getRemoveCount(), i + 1);\n Assert.assertEquals(cache.size(), count - i - 1, \"for i=\" + i);\n }\n\n stats = cache.stats();\n Assert.assertEquals(stats.getRemoveCount(), count);\n Assert.assertEquals(stats.getSize(), 0);\n Assert.assertEquals(stats.getRehashCount(), 2);\n }\n }\n\n\n\n//\n// private String longString()\n// {\n// char[] chars = new char[900];\n// for (int i = 0; i < chars.length; i++)\n// chars[i] = (char) ('A' + i % 26);\n// return new String(chars);\n// }\n//\n//\n @Test(dataProvider = \"hashAlgorithms\", dependsOnMethods = \"testBasics\",\n expectedExceptions = IllegalArgumentException.class,\n expectedExceptionsMessageRegExp = \".*greater than fixed value size.*\")\n public void testPutTooLargeValue(HashAlgorithm hashAlgorithm, boolean useMemoryPool) throws IOException, InterruptedException {\n byte[] key = HashTableTestUtils.randomBytes(8);\n byte[] largeValue = HashTableTestUtils.randomBytes(fixedValueSize + 1);\n\n try (OffHeapHashTable cache = cache(hashAlgorithm, useMemoryPool, 1, -1)) {\n cache.put(key, largeValue);\n }\n }\n\n @Test(dataProvider = \"hashAlgorithms\", dependsOnMethods = \"testBasics\",\n expectedExceptions = IllegalArgumentException.class,\n expectedExceptionsMessageRegExp = \".*exceeds max permitted size of 127\")\n public void testPutTooLargeKey(HashAlgorithm hashAlgorithm, boolean useMemoryPool) throws IOException, InterruptedException {\n byte[] key = HashTableTestUtils.randomBytes(1024);\n byte[] largeValue = HashTableTestUtils.randomBytes(fixedValueSize);\n\n try (OffHeapHashTable cache = cache(hashAlgorithm, useMemoryPool, 1, -1)) {\n cache.put(key, largeValue);\n }\n }\n\n // per-method tests\n\n @Test(dataProvider = \"hashAlgorithms\", dependsOnMethods = \"testBasics\")\n public void testAddOrReplace(HashAlgorithm hashAlgorithm, boolean useMemoryPool) throws Exception\n {\n try (OffHeapHashTable cache = cache(hashAlgorithm, useMemoryPool))\n {\n byte[] oldValue = null;\n for (int i = 0; i < HashTableTestUtils.manyCount; i++)\n assertTrue(cache.addOrReplace(Longs.toByteArray(i), oldValue, HashTableTestUtils\n .randomBytes(fixedValueSize)));\n\n byte[] key = Longs.toByteArray(42);\n byte[] value = cache.get(key);\n byte[] update1 = HashTableTestUtils.randomBytes(fixedValueSize);\n assertTrue(cache.addOrReplace(key, value, update1));\n Assert.assertEquals(cache.get(key), update1);\n\n byte[] update2 = HashTableTestUtils.randomBytes(fixedValueSize);\n assertTrue(cache.addOrReplace(key, update1, update2));\n Assert.assertEquals(cache.get(key), update2);\n Assert.assertFalse(cache.addOrReplace(key, update1, update2));\n Assert.assertEquals(cache.get(key), update2);\n\n cache.remove(key);\n Assert.assertNull(cache.get(key));\n\n byte[] update3 = HashTableTestUtils.randomBytes(fixedValueSize);\n\n // update will fail since the key was removed but old value is non-null.\n Assert.assertFalse(cache.addOrReplace(key, update2, update3));\n Assert.assertNull(cache.get(key));\n }\n }\n\n @Test(dataProvider = \"hashAlgorithms\")\n public void testPutIfAbsent(HashAlgorithm hashAlgorithm, boolean useMemoryPool) throws Exception\n {\n try (OffHeapHashTable cache = cache(hashAlgorithm, useMemoryPool))\n {\n for (int i = 0; i < HashTableTestUtils.manyCount; i++)\n assertTrue(cache.putIfAbsent(Longs.toByteArray(i), HashTableTestUtils.randomBytes(fixedValueSize)));\n\n byte[] key = Longs.toByteArray(HashTableTestUtils.manyCount + 100);\n byte[] value = HashTableTestUtils.randomBytes(fixedValueSize);\n assertTrue(cache.putIfAbsent(key, value));\n Assert.assertEquals(cache.get(key), value);\n Assert.assertFalse(cache.putIfAbsent(key, value));\n }\n }\n\n @Test(dataProvider = \"hashAlgorithms\")\n public void testRemove(HashAlgorithm hashAlgorithm, boolean useMemoryPool) throws Exception\n {\n try (OffHeapHashTable cache = cache(hashAlgorithm, useMemoryPool))\n {\n HashTableTestUtils.fillMany(cache, fixedValueSize);\n\n byte[] key = Longs.toByteArray(HashTableTestUtils.manyCount + 100);\n byte[] value = HashTableTestUtils.randomBytes(fixedValueSize);\n cache.put(key, value);\n Assert.assertEquals(cache.get(key), value);\n cache.remove(key);\n Assert.assertNull(cache.get(key));\n Assert.assertFalse(cache.remove(key));\n\n Random r = new Random();\n for (int i = 0; i < HashTableTestUtils.manyCount; i++)\n cache.remove(Longs.toByteArray(r.nextInt(HashTableTestUtils.manyCount)));\n }\n }\n\n @Test(dataProvider = \"hashAlgorithms\")\n public void testClear(HashAlgorithm hashAlgorithm, boolean useMemoryPool) throws Exception\n {\n try (OffHeapHashTable cache = cache(hashAlgorithm, useMemoryPool))\n {\n List data = new ArrayList<>();\n for (int i = 0; i < 100; i++) {\n data.add(new HashTableTestUtils.KeyValuePair(Longs.toByteArray(i), HashTableTestUtils\n .randomBytes(fixedValueSize)));\n }\n\n data.forEach(kv -> cache.put(kv.key, kv.value));\n data.forEach(kv -> Assert.assertEquals(cache.get(kv.key), kv.value));\n\n assertEquals(cache.size(), 100);\n\n cache.clear();\n assertEquals(cache.size(), 0);\n }\n }\n\n @Test(dataProvider = \"hashAlgorithms\")\n public void testGet_Put(HashAlgorithm hashAlgorithm, boolean useMemoryPool) throws Exception\n {\n try (OffHeapHashTable cache = cache(hashAlgorithm, useMemoryPool))\n {\n byte[] key = Longs.toByteArray(42);\n byte[] value = HashTableTestUtils.randomBytes(fixedValueSize);\n cache.put(key, value);\n assertEquals(cache.get(key), value);\n Assert.assertNull(cache.get(Longs.toByteArray(5)));\n\n byte[] key11 = Longs.toByteArray(11);\n byte[] value11 = HashTableTestUtils.randomBytes(fixedValueSize);\n cache.put(key11, value11);\n Assert.assertEquals(cache.get(key), value);\n Assert.assertEquals(cache.get(key11), value11);\n\n value11 = HashTableTestUtils.randomBytes(fixedValueSize);\n cache.put(key11, value11);\n Assert.assertEquals(cache.get(key), value);\n Assert.assertEquals(cache.get(key11), value11);\n }\n }\n\n @Test(dataProvider = \"hashAlgorithms\")\n public void testContainsKey(HashAlgorithm hashAlgorithm, boolean useMemoryPool) throws Exception\n {\n try (OffHeapHashTable cache = cache(hashAlgorithm, useMemoryPool))\n {\n byte[] key = Longs.toByteArray(42);\n byte[] value = HashTableTestUtils.randomBytes(fixedValueSize);\n cache.put(key, value);\n assertTrue(cache.containsKey(key));\n Assert.assertFalse(cache.containsKey(Longs.toByteArray(11)));\n }\n }\n\n\n @Test(dataProvider = \"hashAlgorithms\")\n public void testGetBucketHistogram(HashAlgorithm hashAlgorithm, boolean useMemoryPool) throws Exception\n {\n try (DoubleCheckOffHeapHashTableImpl cache = cache(hashAlgorithm, useMemoryPool))\n {\n List data = new ArrayList<>();\n for (int i = 0; i < 100; i++) {\n data.add(new HashTableTestUtils.KeyValuePair(Longs.toByteArray(i), HashTableTestUtils\n .randomBytes(fixedValueSize)));\n }\n\n data.forEach(kv -> cache.put(kv.key, kv.value));\n\n Assert.assertEquals(cache.stats().getSize(), 100);\n\n EstimatedHistogram hProd = cache.prod.getBucketHistogram();\n Assert.assertEquals(hProd.count(), sum(cache.prod.hashTableSizes()));\n long[] offsets = hProd.getBucketOffsets();\n Assert.assertEquals(offsets.length, 3);\n Assert.assertEquals(offsets[0], -1);\n Assert.assertEquals(offsets[1], 0);\n Assert.assertEquals(offsets[2], 1);\n // hProd.log(LoggerFactory.getLogger(CrossCheckTest.class));\n // System.out.println(Arrays.toString(offsets));\n Assert.assertEquals(hProd.min(), 0);\n Assert.assertEquals(hProd.max(), 1);\n }\n }\n\n private static int sum(int[] ints)\n {\n int r = 0;\n for (int i : ints)\n r += i;\n return r;\n }\n\n @Test(dataProvider = \"hashAlgorithms\")\n public void testResetStatistics(HashAlgorithm hashAlgorithm, boolean useMemoryPool) throws IOException\n {\n try (OffHeapHashTable cache = cache(hashAlgorithm, useMemoryPool))\n {\n for (int i = 0; i < 100; i++)\n cache.put(Longs.toByteArray(i), HashTableTestUtils.randomBytes(fixedValueSize));\n\n for (int i = 0; i < 30; i++)\n cache.put(Longs.toByteArray(i), HashTableTestUtils.randomBytes(fixedValueSize));\n\n for (int i = 0; i < 50; i++)\n cache.get(Longs.toByteArray(i));\n\n for (int i = 100; i < 120; i++)\n cache.get(Longs.toByteArray(i));\n\n for (int i = 0; i < 25; i++)\n cache.remove(Longs.toByteArray(i));\n\n OffHeapHashTableStats stats = cache.stats();\n Assert.assertEquals(stats.getPutAddCount(), 100);\n Assert.assertEquals(stats.getPutReplaceCount(), 30);\n Assert.assertEquals(stats.getHitCount(), 50);\n Assert.assertEquals(stats.getMissCount(), 20);\n Assert.assertEquals(stats.getRemoveCount(), 25);\n\n cache.resetStatistics();\n\n stats = cache.stats();\n Assert.assertEquals(stats.getPutAddCount(), 0);\n Assert.assertEquals(stats.getPutReplaceCount(), 0);\n Assert.assertEquals(stats.getHitCount(), 0);\n Assert.assertEquals(stats.getMissCount(), 0);\n Assert.assertEquals(stats.getRemoveCount(), 0);\n }\n }\n}\n" }, { "path": "src/test/java/com/oath/halodb/DBDirectoryTest.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport org.hamcrest.MatcherAssert;\nimport org.hamcrest.Matchers;\nimport org.testng.Assert;\nimport org.testng.annotations.AfterMethod;\nimport org.testng.annotations.BeforeMethod;\nimport org.testng.annotations.Test;\n\nimport java.io.File;\nimport java.io.FileWriter;\nimport java.io.IOException;\nimport java.io.PrintWriter;\nimport java.nio.file.Path;\nimport java.nio.file.Paths;\nimport java.util.List;\nimport java.util.stream.Collectors;\nimport java.util.stream.Stream;\n\npublic class DBDirectoryTest {\n\n private static final File directory = Paths.get(\"tmp\", \"DBDirectoryTest\").toFile();\n private DBDirectory dbDirectory;\n\n private static Integer[] dataFileIds = {7, 12, 1, 8, 10};\n private static Integer[] tombstoneFileIds = {21, 13, 12};\n\n @Test\n public void testListIndexFiles() {\n List actual = dbDirectory.listIndexFiles();\n List expected = Stream.of(dataFileIds).sorted().collect(Collectors.toList());\n Assert.assertEquals(actual, expected);\n Assert.assertEquals(actual.size(), dataFileIds.length);\n }\n\n @Test\n public void testListDataFiles() {\n File[] files = dbDirectory.listDataFiles();\n List actual = Stream.of(files).map(File::getName).collect(Collectors.toList());\n List expected = Stream.of(dataFileIds).map(i -> i + HaloDBFile.DATA_FILE_NAME).collect(Collectors.toList());\n MatcherAssert.assertThat(actual, Matchers.containsInAnyOrder(expected.toArray()));\n Assert.assertEquals(actual.size(), dataFileIds.length);\n }\n\n @Test\n public void testListTombstoneFiles() {\n File[] files = dbDirectory.listTombstoneFiles();\n List actual = Stream.of(files).map(File::getName).collect(Collectors.toList());\n List expected = Stream.of(tombstoneFileIds).sorted().map(i -> i + TombstoneFile.TOMBSTONE_FILE_NAME).collect(Collectors.toList());\n\n Assert.assertEquals(actual, expected);\n Assert.assertEquals(actual.size(), tombstoneFileIds.length);\n }\n\n @Test\n public void testSyncMetaDataNoError() throws IOException {\n dbDirectory.syncMetaData();\n }\n\n @BeforeMethod\n public void createDirectory() throws IOException {\n dbDirectory = DBDirectory.open(directory);\n\n Path directoryPath = dbDirectory.getPath();\n for (int i : dataFileIds) {\n try(PrintWriter writer = new PrintWriter(new FileWriter(\n directoryPath.resolve(i + IndexFile.INDEX_FILE_NAME).toString()))) {\n writer.append(\"test\");\n }\n\n try(PrintWriter writer = new PrintWriter(new FileWriter(\n directoryPath.resolve(i + HaloDBFile.DATA_FILE_NAME).toString()))) {\n writer.append(\"test\");\n }\n }\n\n // repair file, should be skipped. \n try(PrintWriter writer = new PrintWriter(new FileWriter(\n directoryPath.resolve(10000 + HaloDBFile.DATA_FILE_NAME + \".repair\").toString()))) {\n writer.append(\"test\");\n }\n\n for (int i : tombstoneFileIds) {\n try(PrintWriter writer = new PrintWriter(new FileWriter(\n directoryPath.resolve(i + TombstoneFile.TOMBSTONE_FILE_NAME).toString()))) {\n writer.append(\"test\");\n }\n }\n\n // repair file, should be skipped.\n try(PrintWriter writer = new PrintWriter(new FileWriter(\n directoryPath.resolve(20000 + TombstoneFile.TOMBSTONE_FILE_NAME + \".repair\").toString()))) {\n writer.append(\"test\");\n }\n }\n\n @AfterMethod\n public void deleteDirectory() throws IOException {\n dbDirectory.close();\n TestUtils.deleteDirectory(directory);\n }\n}\n" }, { "path": "src/test/java/com/oath/halodb/DBMetaDataTest.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport org.testng.Assert;\nimport org.testng.annotations.AfterMethod;\nimport org.testng.annotations.BeforeMethod;\nimport org.testng.annotations.Test;\n\nimport java.io.File;\nimport java.io.IOException;\nimport java.nio.file.Files;\nimport java.nio.file.Path;\nimport java.nio.file.Paths;\n\npublic class DBMetaDataTest {\n\n private static final File directory = Paths.get(\"tmp\", \"DBMetaDataTest\", \"testDBMetaData\").toFile();\n private DBDirectory dbDirectory;\n\n @Test\n public void testDBMetaData() throws IOException {\n Path metaDataFile = dbDirectory.getPath().resolve(DBMetaData.METADATA_FILE_NAME);\n\n // confirm that the file doesn't exist.\n Assert.assertFalse(Files.exists(metaDataFile));\n\n DBMetaData metaData = new DBMetaData(dbDirectory);\n metaData.loadFromFileIfExists();\n\n // file has not yet been created, return default values.\n Assert.assertEquals(metaData.getVersion(), 0);\n Assert.assertEquals(metaData.getMaxFileSize(), 0);\n Assert.assertFalse(metaData.isOpen());\n Assert.assertEquals(metaData.getSequenceNumber(), 0);\n Assert.assertFalse(metaData.isIOError());\n\n metaData.setVersion(Versions.CURRENT_META_FILE_VERSION);\n metaData.setOpen(true);\n metaData.setSequenceNumber(100);\n metaData.setIOError(false);\n metaData.setMaxFileSize(100);\n metaData.storeToFile();\n\n // confirm that the file has been created.\n Assert.assertTrue(Files.exists(metaDataFile));\n\n // load again to read stored values.\n metaData = new DBMetaData(dbDirectory);\n metaData.loadFromFileIfExists();\n\n Assert.assertEquals(metaData.getVersion(), Versions.CURRENT_META_FILE_VERSION);\n Assert.assertTrue(metaData.isOpen());\n Assert.assertEquals(metaData.getSequenceNumber(), 100);\n Assert.assertFalse(metaData.isIOError());\n Assert.assertEquals(metaData.getMaxFileSize(), 100);\n\n metaData.setVersion(Versions.CURRENT_META_FILE_VERSION + 10);\n metaData.setOpen(false);\n metaData.setSequenceNumber(Long.MAX_VALUE);\n metaData.setIOError(true);\n metaData.setMaxFileSize(1024);\n metaData.storeToFile();\n\n // load again to read stored values.\n metaData = new DBMetaData(dbDirectory);\n metaData.loadFromFileIfExists();\n\n Assert.assertEquals(metaData.getVersion(), Versions.CURRENT_META_FILE_VERSION + 10);\n Assert.assertFalse(metaData.isOpen());\n Assert.assertEquals(metaData.getSequenceNumber(), Long.MAX_VALUE);\n Assert.assertTrue(metaData.isIOError());\n Assert.assertEquals(metaData.getMaxFileSize(), 1024);\n }\n\n @Test\n public void testCheckSum() throws IOException {\n DBMetaData metaData = new DBMetaData(dbDirectory);\n metaData.loadFromFileIfExists();\n\n metaData.setVersion(Versions.CURRENT_META_FILE_VERSION);\n metaData.setOpen(true);\n metaData.setSequenceNumber(100);\n metaData.setIOError(false);\n metaData.setMaxFileSize(100);\n metaData.storeToFile();\n\n metaData = new DBMetaData(dbDirectory);\n metaData.loadFromFileIfExists();\n\n Assert.assertTrue(metaData.isValid());\n\n }\n\n @BeforeMethod\n public void createDirectory() throws IOException {\n dbDirectory = DBDirectory.open(directory);\n }\n\n @AfterMethod\n public void deleteDirectory() throws IOException {\n dbDirectory.close();\n TestUtils.deleteDirectory(directory);\n }\n\n}\n" }, { "path": "src/test/java/com/oath/halodb/DBRepairTest.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport org.testng.Assert;\nimport org.testng.annotations.Test;\n\nimport java.io.File;\nimport java.io.IOException;\nimport java.nio.file.attribute.FileTime;\nimport java.util.ArrayList;\nimport java.util.List;\nimport java.util.stream.Collectors;\nimport java.util.stream.IntStream;\n\npublic class DBRepairTest extends TestBase {\n\n @Test(dataProvider = \"Options\")\n public void testRepairDB(HaloDBOptions options) throws HaloDBException, IOException {\n String directory = TestUtils.getTestDirectory(\"DBRepairTest\", \"testRepairDB\");\n\n options.setMaxFileSize(1024 * 1024);\n options.setCompactionDisabled(true);\n\n HaloDB db = getTestDB(directory, options);\n int noOfRecords = 5 * 1024 + 512; // 5 files with 1024 records and 1 with 512 records. \n\n List records = TestUtils.insertRandomRecordsOfSize(db, noOfRecords, 1024-Record.Header.HEADER_SIZE);\n\n // delete half the records.\n for (int i = 0; i < noOfRecords; i++) {\n if (i % 2 == 0) {\n db.delete(records.get(i).getKey());\n }\n }\n\n FileTime latestDataFileCreatedTime =\n TestUtils.getFileCreationTime(TestUtils.getLatestDataFile(directory).get());\n FileTime latestTombstoneFileCreationTime =\n TestUtils.getFileCreationTime(FileUtils.listTombstoneFiles(new File(directory))[0]);\n\n db.close();\n\n // trick the db to think that there was an unclean shutdown.\n DBMetaData dbMetaData = new DBMetaData(dbDirectory);\n dbMetaData.setOpen(true);\n dbMetaData.storeToFile();\n\n // wait for a second so that the new file will have a different create at time.\n try {\n Thread.sleep(1000);\n } catch (InterruptedException e) {\n }\n\n db = getTestDBWithoutDeletingFiles(directory, options);\n\n // latest file should have been repaired and replaced.\n Assert.assertNotEquals(\n TestUtils.getFileCreationTime(TestUtils.getLatestDataFile(directory).get()),\n latestDataFileCreatedTime\n );\n Assert.assertNotEquals(\n TestUtils.getFileCreationTime(FileUtils.listTombstoneFiles(new File(directory))[0]),\n latestTombstoneFileCreationTime\n );\n\n Assert.assertEquals(db.size(), noOfRecords/2);\n for (int i = 0; i < noOfRecords; i++) {\n if (i % 2 == 0) {\n Assert.assertNull(db.get(records.get(i).getKey()));\n }\n else {\n Record r = records.get(i);\n Assert.assertEquals(db.get(r.getKey()), r.getValue());\n }\n }\n }\n\n @Test(dataProvider = \"Options\")\n public void testRepairDBWithCompaction(HaloDBOptions options) throws HaloDBException, InterruptedException, IOException {\n String directory = TestUtils.getTestDirectory(\"DBRepairTest\", \"testRepairDBWithCompaction\");\n\n options.setMaxFileSize(1024 * 1024);\n options.setCompactionThresholdPerFile(0.5);\n HaloDB db = getTestDB(directory, options);\n int noOfRecords = 10 * 1024 + 512;\n\n List records = TestUtils.insertRandomRecordsOfSize(db, noOfRecords, 1024-Record.Header.HEADER_SIZE);\n List toUpdate = IntStream.range(0, noOfRecords).filter(i -> i%2==0).mapToObj(i -> records.get(i)).collect(Collectors.toList());\n List updatedRecords = TestUtils.updateRecords(db, toUpdate);\n for (int i = 0; i < updatedRecords.size(); i++) {\n records.set(i * 2, updatedRecords.get(i));\n }\n\n TestUtils.waitForCompactionToComplete(db);\n\n FileTime latestDataFileCreatedTime =\n TestUtils.getFileCreationTime(TestUtils.getLatestDataFile(directory).get());\n\n db.close();\n\n // trick the db to think that there was an unclean shutdown.\n DBMetaData dbMetaData = new DBMetaData(dbDirectory);\n dbMetaData.setOpen(true);\n dbMetaData.storeToFile();\n\n // wait for a second so that the new file will have a different created at time.\n try {\n Thread.sleep(1000);\n } catch (InterruptedException e) {\n }\n\n db = getTestDBWithoutDeletingFiles(directory, options);\n\n // latest file should have been repaired and replaced.\n Assert.assertNotEquals(\n TestUtils.getFileCreationTime(TestUtils.getLatestDataFile(directory).get()),\n latestDataFileCreatedTime\n );\n\n Assert.assertEquals(db.size(), noOfRecords);\n for (Record r : records) {\n Assert.assertEquals(db.get(r.getKey()), r.getValue());\n }\n }\n\n @Test\n public void testRepairWithMultipleTombstoneFiles() throws HaloDBException, IOException {\n String directory = TestUtils.getTestDirectory(\"DBRepairTest\", \"testRepairWithMultipleTombstoneFiles\");\n\n HaloDBOptions options = new HaloDBOptions();\n options.setCompactionDisabled(true);\n options.setMaxFileSize(320);\n HaloDB db = getTestDB(directory, options);\n\n int noOfTombstonesPerFile = 10;\n int noOfFiles = 3;\n int noOfRecords = noOfTombstonesPerFile * noOfFiles;\n\n // Since keyLength was 19 tombstone entry is 32 bytes.\n int keyLength = 19;\n int valueLength = 24;\n\n List records = new ArrayList<>();\n for (int i = 0; i < noOfRecords; i++) {\n Record r = new Record(TestUtils.generateRandomByteArray(keyLength), TestUtils.generateRandomByteArray(valueLength));\n records.add(r);\n db.put(r.getKey(), r.getValue());\n }\n\n for (Record r : records) {\n db.delete(r.getKey());\n }\n\n File[] tombstoneFiles = FileUtils.listTombstoneFiles(new File(directory));\n\n FileTime latestDataFileCreatedTime =\n TestUtils.getFileCreationTime(TestUtils.getLatestDataFile(directory).get());\n FileTime latestTombstoneFileCreationTime =\n TestUtils.getFileCreationTime(tombstoneFiles[tombstoneFiles.length-1]);\n\n db.close();\n\n // trick the db to think that there was an unclean shutdown.\n DBMetaData dbMetaData = new DBMetaData(dbDirectory);\n dbMetaData.setOpen(true);\n dbMetaData.storeToFile();\n\n try {\n Thread.sleep(1000);\n } catch (InterruptedException e) {\n }\n\n db = getTestDBWithoutDeletingFiles(directory, options);\n\n // latest file should have been repaired and replaced.\n Assert.assertNotEquals(\n TestUtils.getFileCreationTime(TestUtils.getLatestDataFile(directory).get()),\n latestDataFileCreatedTime\n );\n Assert.assertNotEquals(\n TestUtils.getFileCreationTime(tombstoneFiles[tombstoneFiles.length-1]),\n latestTombstoneFileCreationTime\n );\n\n\n // other two tombstone files should still be there\n Assert.assertTrue(tombstoneFiles[0].exists());\n Assert.assertTrue(tombstoneFiles[1].exists());\n\n Assert.assertEquals(db.size(), 0);\n for (int i = 0; i < noOfRecords; i++) {\n Assert.assertNull(db.get(records.get(i).getKey()));\n }\n }\n}\n" }, { "path": "src/test/java/com/oath/halodb/DataConsistencyDB.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport java.nio.ByteBuffer;\nimport java.util.Arrays;\nimport java.util.Map;\nimport java.util.concurrent.ConcurrentHashMap;\nimport java.util.concurrent.locks.ReentrantReadWriteLock;\n\n/**\n * Holds an instance of HaloDB and Java's ConcurrentHashMap.\n * Tests will use this to insert data into both and ensure that\n * the data in HaloDB is correct. \n */\n\nclass DataConsistencyDB {\n private static final Logger logger = LoggerFactory.getLogger(DataConsistencyDB.class);\n\n //TODO: allocate this off-heap.\n private final Map javaMap = new ConcurrentHashMap<>();\n private final HaloDB haloDB;\n\n private int numberOfLocks = 100;\n\n private final ReentrantReadWriteLock[] locks;\n\n DataConsistencyDB(HaloDB haloDB, int noOfRecords) {\n this.haloDB = haloDB;\n\n locks = new ReentrantReadWriteLock[numberOfLocks];\n for (int i = 0; i < numberOfLocks; i++) {\n locks[i] = new ReentrantReadWriteLock();\n }\n }\n\n void put(int keyIndex, ByteBuffer keyBuf, byte[] value) throws HaloDBException {\n ReentrantReadWriteLock lock = locks[keyIndex%numberOfLocks];\n try {\n lock.writeLock().lock();\n javaMap.put(keyBuf, value);\n haloDB.put(keyBuf.array(), value);\n }\n finally {\n lock.writeLock().unlock();\n }\n }\n\n // return -1 if values don't match.\n int compareValues(int keyIndex, ByteBuffer keyBuf) throws HaloDBException {\n ReentrantReadWriteLock lock = locks[keyIndex%numberOfLocks];\n try {\n lock.readLock().lock();\n return checkValues(keyIndex, keyBuf, haloDB);\n }\n finally {\n lock.readLock().unlock();\n }\n }\n\n boolean checkSize() {\n return haloDB.size() == javaMap.size();\n }\n\n void delete(int keyIndex, ByteBuffer keyBuf) throws HaloDBException {\n ReentrantReadWriteLock lock = locks[keyIndex%numberOfLocks];\n try {\n lock.writeLock().lock();\n javaMap.remove(keyBuf);\n haloDB.delete(keyBuf.array());\n }\n finally {\n lock.writeLock().unlock();\n }\n }\n\n boolean iterateAndCheck(HaloDB db) {\n if (db.size() != javaMap.size()) {\n logger.error(\"Size don't match {} != {}\", db.size(), javaMap.size());\n return false;\n }\n\n for (Map.Entry entry : javaMap.entrySet()) {\n try {\n if (!Arrays.equals(entry.getValue(), db.get(entry.getKey().array()))) {\n return false;\n }\n\n } catch (HaloDBException e) {\n logger.error(\"Error while iterating\", e);\n return false;\n }\n }\n return true;\n }\n\n // return -1 if values don't match.\n private int checkValues(long key, ByteBuffer keyBuf, HaloDB haloDB) throws HaloDBException {\n byte[] mapValue = javaMap.get(keyBuf);\n byte[] dbValue = haloDB.get(keyBuf.array());\n if (Arrays.equals(mapValue, dbValue))\n return dbValue == null ? 0 : dbValue.length;\n\n if (mapValue == null) {\n logger.error(\"Map value is null for key {} of length {} but HaloDB value has version {}\",\n key, keyBuf.remaining(), DataConsistencyTest.getVersionFromValue(dbValue));\n }\n else if (dbValue == null) {\n logger.error(\"HaloDB value is null for key {} of length {} but Map value has version {}\",\n key, keyBuf.remaining(), DataConsistencyTest.getVersionFromValue(mapValue));\n }\n else {\n logger.error(\"HaloDB value for key {} has version {} of length {} but map value version is {}\",\n key, keyBuf.remaining(), DataConsistencyTest.getVersionFromValue(dbValue), DataConsistencyTest\n .getVersionFromValue(mapValue));\n }\n\n return -1;\n }\n\n boolean containsKey(byte[] key) throws HaloDBException {\n return haloDB.get(key) != null;\n }\n}\n" }, { "path": "src/test/java/com/oath/halodb/DataConsistencyTest.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport com.google.common.primitives.Longs;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\nimport org.testng.Assert;\nimport org.testng.annotations.BeforeMethod;\nimport org.testng.annotations.Test;\n\nimport java.nio.ByteBuffer;\nimport java.util.HashSet;\nimport java.util.Random;\nimport java.util.Set;\n\npublic class DataConsistencyTest extends TestBase {\n private static final Logger logger = LoggerFactory.getLogger(DataConsistencyTest.class);\n\n private final Object lock = new Object();\n private volatile boolean insertionComplete;\n private volatile boolean updatesComplete;\n private volatile boolean foundNonMatchingValue;\n\n private static final int fixedKeySize = 16;\n private static final int maxValueSize = 100;\n\n private static final int noOfRecords = 100_000;\n private static final int noOfTransactions = 1_000_000;\n\n private ByteBuffer[] keys;\n\n private RandomDataGenerator randDataGenerator;\n private Random random = new Random();\n\n private HaloDB haloDB;\n\n @BeforeMethod\n public void init() {\n insertionComplete = false;\n updatesComplete = false;\n foundNonMatchingValue = false;\n keys = new ByteBuffer[noOfRecords];\n randDataGenerator = new RandomDataGenerator();\n }\n\n @Test(dataProvider = \"Options\")\n public void testConcurrentReadAndUpdates(HaloDBOptions options) throws HaloDBException, InterruptedException {\n\n String directory = TestUtils.getTestDirectory(\"DataConsistencyCheck\", \"testConcurrentReadAndUpdates\");\n\n options.setMaxFileSize(10 * 1024);\n options.setCompactionThresholdPerFile(0.1);\n options.setFixedKeySize(fixedKeySize);\n\n options.setNumberOfRecords(2 * noOfRecords);\n\n haloDB = getTestDB(directory, options);\n DataConsistencyDB db = new DataConsistencyDB(haloDB, noOfRecords);\n\n Writer writer = new Writer(db);\n writer.start();\n\n synchronized (lock) {\n while (!insertionComplete) {\n lock.wait();\n }\n }\n\n long start = System.currentTimeMillis();\n Reader[] readers = new Reader[10];\n\n for (int i = 0; i < readers.length; i++) {\n readers[i] = new Reader(db);\n readers[i].start();\n }\n\n writer.join();\n long totalReads = 0, totalReadSize = 0;\n for (Reader reader : readers) {\n reader.join();\n totalReads += reader.readCount;\n totalReadSize += reader.readSize;\n }\n long time = (System.currentTimeMillis() - start)/1000;\n\n Assert.assertFalse(foundNonMatchingValue);\n Assert.assertTrue(db.checkSize());\n\n haloDB.close();\n\n logger.info(\"Iterating and checking ...\");\n HaloDB openAgainDB = getTestDBWithoutDeletingFiles(directory, options);\n TestUtils.waitForCompactionToComplete(openAgainDB);\n Assert.assertTrue(db.iterateAndCheck(openAgainDB));\n\n logger.info(\"Completed {} updates\", writer.updateCount);\n logger.info(\"Completed {} deletes\", writer.deleteCount);\n logger.info(\"Completed {} reads\", totalReads);\n logger.info(\"Reads per second {}. {} MB/second\", totalReads/time, totalReadSize/1024/1024/time);\n logger.info(\"Writes per second {}. {} KB/second\", noOfTransactions/time, writer.totalWriteSize/1024/time);\n logger.info(\"Compaction rate {} KB/second\", haloDB.stats().getCompactionRateSinceBeginning()/1024);\n }\n\n class Writer extends Thread {\n\n DataConsistencyDB db;\n long updateCount = 0;\n long deleteCount = 0;\n volatile long totalWriteSize = 0;\n Set deletedKeys = new HashSet<>(50_000);\n\n Writer(DataConsistencyDB db) {\n this.db = db;\n setPriority(MAX_PRIORITY);\n }\n\n @Override\n public void run() {\n Random random = new Random();\n\n try {\n for (int i = 0; i < noOfRecords; i++) {\n try {\n byte[] key = randDataGenerator.getData(getRandomKeyLength());\n while (db.containsKey(key)) {\n key = randDataGenerator.getData(getRandomKeyLength());\n }\n\n keys[i] = ByteBuffer.wrap(key);\n // we need at least 8 bytes for the version.\n int size = random.nextInt(maxValueSize) + 9;\n db.put(i, keys[i], generateRandomValueWithVersion(updateCount, size));\n } catch (HaloDBException e) {\n throw new RuntimeException(e);\n }\n }\n } finally {\n synchronized (lock) {\n insertionComplete = true;\n lock.notify();\n }\n }\n\n try {\n while (!foundNonMatchingValue && updateCount < noOfTransactions) {\n int k = random.nextInt(noOfRecords);\n int size = random.nextInt(maxValueSize) + 9;\n updateCount++;\n try {\n if (updateCount % 2 == 0) {\n db.delete(k, keys[k]);\n deleteCount++;\n deletedKeys.add(k);\n if (deletedKeys.size() == 50_000) {\n int keyToAdd = deletedKeys.iterator().next();\n db.put(keyToAdd, keys[keyToAdd], generateRandomValueWithVersion(updateCount, size));\n totalWriteSize += size;\n deletedKeys.remove(keyToAdd);\n }\n }\n else {\n db.put(k, keys[k], generateRandomValueWithVersion(updateCount, size));\n totalWriteSize += size;\n }\n } catch (HaloDBException e) {\n throw new RuntimeException(e);\n }\n\n if (updateCount > 0 && updateCount % 500_000 == 0) {\n logger.info(\"Completed {} updates\", updateCount);\n }\n\n }\n } finally {\n updatesComplete = true;\n }\n }\n }\n\n class Reader extends Thread {\n\n DataConsistencyDB db;\n volatile long readCount = 0;\n volatile long readSize = 0;\n\n Reader(DataConsistencyDB db) {\n this.db = db;\n setPriority(MIN_PRIORITY);\n }\n\n @Override\n public void run() {\n Random random = new Random();\n while (!updatesComplete) {\n int i = random.nextInt(noOfRecords);\n try {\n int valueSize = db.compareValues(i, keys[i]);\n readCount++;\n if (valueSize == -1) {\n foundNonMatchingValue = true;\n }\n Assert.assertNotEquals(valueSize, -1, \"Values don't match for key \" + i);\n readSize += valueSize;\n } catch (HaloDBException e) {\n throw new RuntimeException(e);\n }\n }\n }\n }\n\n private int getRandomKeyLength() {\n return random.nextInt(fixedKeySize) + 1;\n }\n\n private byte[] generateRandomValueWithVersion(long version, int size) {\n byte[] value = randDataGenerator.getData(size);\n System.arraycopy(Longs.toByteArray(version), 0, value, size - 8, 8);\n return value;\n }\n\n static long getVersionFromValue(byte[] value) {\n byte[] v = new byte[8];\n System.arraycopy(value, value.length-8, v, 0, 8);\n return Longs.fromByteArray(v);\n }\n}" }, { "path": "src/test/java/com/oath/halodb/DoubleCheckOffHeapHashTableImpl.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\n// This code is a derivative work heavily modified from the OHC project. See NOTICE file for copyright and license.\n\npackage com.oath.halodb;\n\nimport com.google.common.primitives.Longs;\nimport com.oath.halodb.histo.EstimatedHistogram;\n\nimport org.testng.Assert;\n\nimport java.io.IOException;\n\n/**\n * Test code that contains an instance of the production and check {@link OffHeapHashTable}\n * implementations {@link OffHeapHashTableImpl} and\n * {@link CheckOffHeapHashTable}.\n */\npublic class DoubleCheckOffHeapHashTableImpl implements OffHeapHashTable\n{\n public final OffHeapHashTable prod;\n public final OffHeapHashTable check;\n\n public DoubleCheckOffHeapHashTableImpl(OffHeapHashTableBuilder builder)\n {\n this.prod = builder.build();\n this.check = new CheckOffHeapHashTable<>(builder);\n }\n\n public boolean put(byte[] key, V value)\n {\n boolean rProd = prod.put(key, value);\n boolean rCheck = check.put(key, value);\n Assert.assertEquals(rProd, rCheck, \"for key='\" + key + '\\'');\n return rProd;\n }\n\n public boolean addOrReplace(byte[] key, V old, V value)\n {\n boolean rProd = prod.addOrReplace(key, old, value);\n boolean rCheck = check.addOrReplace(key, old, value);\n Assert.assertEquals(rProd, rCheck, \"for key='\" + key + '\\'');\n return rProd;\n }\n\n public boolean putIfAbsent(byte[] k, V v)\n {\n boolean rProd = prod.putIfAbsent(k, v);\n boolean rCheck = check.putIfAbsent(k, v);\n Assert.assertEquals(rProd, rCheck, \"for key='\" + k + '\\'');\n return rProd;\n }\n\n public boolean putIfAbsent(byte[] key, V value, long expireAt)\n {\n throw new UnsupportedOperationException();\n }\n\n public boolean put(byte[] key, V value, long expireAt)\n {\n throw new UnsupportedOperationException();\n }\n\n public boolean remove(byte[] key)\n {\n boolean rProd = prod.remove(key);\n boolean rCheck = check.remove(key);\n Assert.assertEquals(rCheck, rProd, \"for key='\" + key + '\\'');\n return rProd;\n }\n\n public void clear()\n {\n prod.clear();\n check.clear();\n }\n\n public V get(byte[] key)\n {\n V rProd = prod.get(key);\n V rCheck = check.get(key);\n Assert.assertEquals(rProd, rCheck, \"for key='\" + Longs.fromByteArray(key) + '\\'');\n return rProd;\n }\n\n public boolean containsKey(byte[] key)\n {\n boolean rProd = prod.containsKey(key);\n boolean rCheck = check.containsKey(key);\n Assert.assertEquals(rProd, rCheck, \"for key='\" + key + '\\'');\n return rProd;\n }\n\n public void resetStatistics()\n {\n prod.resetStatistics();\n check.resetStatistics();\n }\n\n public long size()\n {\n long rProd = prod.size();\n long rCheck = check.size();\n Assert.assertEquals(rProd, rCheck);\n return rProd;\n }\n\n public int[] hashTableSizes()\n {\n return prod.hashTableSizes();\n }\n\n @Override\n public SegmentStats[] perSegmentStats() {\n SegmentStats[] rProd = prod.perSegmentStats();\n SegmentStats[] rCheck = check.perSegmentStats();\n Assert.assertEquals(rProd, rCheck);\n return rProd;\n }\n\n public EstimatedHistogram getBucketHistogram()\n {\n return prod.getBucketHistogram();\n }\n\n public int segments()\n {\n int rProd = prod.segments();\n int rCheck = check.segments();\n Assert.assertEquals(rProd, rCheck);\n return rProd;\n }\n\n public float loadFactor()\n {\n float rProd = prod.loadFactor();\n float rCheck = check.loadFactor();\n Assert.assertEquals(rProd, rCheck);\n return rProd;\n }\n\n public OffHeapHashTableStats stats()\n {\n OffHeapHashTableStats rProd = prod.stats();\n OffHeapHashTableStats rCheck = check.stats();\n Assert.assertEquals(rProd, rCheck);\n return rProd;\n }\n\n public void close() throws IOException\n {\n prod.close();\n check.close();\n }\n}\n" }, { "path": "src/test/java/com/oath/halodb/FileUtilsTest.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport java.nio.file.Files;\nimport java.nio.file.Path;\nimport org.hamcrest.MatcherAssert;\nimport org.hamcrest.Matchers;\nimport org.testng.Assert;\nimport org.testng.annotations.AfterMethod;\nimport org.testng.annotations.BeforeMethod;\nimport org.testng.annotations.Test;\n\nimport java.io.File;\nimport java.io.FileWriter;\nimport java.io.IOException;\nimport java.io.PrintWriter;\nimport java.nio.file.Paths;\nimport java.util.List;\nimport java.util.stream.Collectors;\nimport java.util.stream.Stream;\n\npublic class FileUtilsTest {\n\n private String directory = TestUtils.getTestDirectory(\"FileUtilsTest\");\n\n private Integer[] fileIds = {7, 12, 1, 8, 10};\n\n private List indexFileNames =\n Stream.of(fileIds)\n .map(i -> Paths.get(directory).resolve(i + IndexFile.INDEX_FILE_NAME).toString())\n .collect(Collectors.toList());\n\n\n private List dataFileNames =\n Stream.of(fileIds)\n .map(i -> Paths.get(directory).resolve(i + HaloDBFile.DATA_FILE_NAME).toString())\n .collect(Collectors.toList());\n\n\n private List dataFileNamesRepair =\n Stream.of(fileIds)\n .map(i -> Paths.get(directory).resolve(i + HaloDBFile.DATA_FILE_NAME + \".repair\").toString())\n .collect(Collectors.toList());\n\n\n private List tombstoneFileNames =\n Stream.of(fileIds)\n .map(i -> Paths.get(directory).resolve(i + TombstoneFile.TOMBSTONE_FILE_NAME).toString())\n .collect(Collectors.toList());\n\n\n @BeforeMethod\n public void createDirectory() throws IOException {\n TestUtils.deleteDirectory(new File(directory));\n FileUtils.createDirectoryIfNotExists(new File(directory));\n\n for (String f : indexFileNames) {\n try(PrintWriter writer = new PrintWriter(new FileWriter(f))) {\n writer.append(\"test\");\n }\n }\n\n for (String f : dataFileNames) {\n try(PrintWriter writer = new PrintWriter(new FileWriter(f))) {\n writer.append(\"test\");\n }\n }\n\n for (String f : dataFileNamesRepair) {\n try(PrintWriter writer = new PrintWriter(new FileWriter(f))) {\n writer.append(\"test\");\n }\n }\n\n for (String f : tombstoneFileNames) {\n try(PrintWriter writer = new PrintWriter(new FileWriter(f))) {\n writer.append(\"test\");\n }\n }\n }\n\n @AfterMethod\n public void deleteDirectory() throws IOException {\n TestUtils.deleteDirectory(new File(directory));\n }\n\n @Test\n public void testListIndexFiles() {\n List actual = FileUtils.listIndexFiles(new File(directory));\n\n List expected = Stream.of(fileIds).sorted().collect(Collectors.toList());\n Assert.assertEquals(actual, expected);\n }\n\n @Test\n public void testListDataFiles() {\n File[] files = FileUtils.listDataFiles(new File(directory));\n List actual = Stream.of(files).map(File::getName).collect(Collectors.toList());\n List expected = Stream.of(fileIds).map(i -> i + HaloDBFile.DATA_FILE_NAME).collect(Collectors.toList());\n MatcherAssert.assertThat(actual, Matchers.containsInAnyOrder(expected.toArray()));\n }\n\n @Test\n public void testListTombstoneFiles() {\n File[] files = FileUtils.listTombstoneFiles(new File(directory));\n List actual = Stream.of(files).map(File::getName).collect(Collectors.toList());\n List expected = Stream.of(fileIds).sorted().map(i -> i + TombstoneFile.TOMBSTONE_FILE_NAME).collect(Collectors.toList());\n\n Assert.assertEquals(actual, expected);\n }\n\n @Test\n public void testDirectoryCreateAndDelete() throws IOException {\n Path parentDir = TestUtils.getTestDirectoryPath(\"FileUtilsTest\", \"DirectoryCreateAndDelete\");\n FileUtils.createDirectoryIfNotExists(parentDir.toFile());\n\n Path subDir = TestUtils.getTestDirectoryPath(\"FileUtilsTest\", \"DirectoryCreateAndDelete\", \"subDir\");\n FileUtils.createDirectoryIfNotExists(subDir.toFile());\n\n\n FileUtils.deleteDirectory(parentDir.toFile());\n\n Assert.assertFalse(Files.exists(parentDir));\n }\n\n}\n" }, { "path": "src/test/java/com/oath/halodb/HaloDBCompactionTest.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport com.google.common.primitives.Longs;\n\nimport org.testng.Assert;\nimport org.testng.annotations.Test;\n\nimport java.io.File;\nimport java.util.ArrayList;\nimport java.util.List;\nimport java.util.stream.Collectors;\nimport java.util.stream.IntStream;\n\npublic class HaloDBCompactionTest extends TestBase {\n\n private final int recordSize = 1024;\n private final int numberOfFiles = 8;\n private final int recordsPerFile = 10;\n private final int numberOfRecords = numberOfFiles * recordsPerFile;\n\n @Test(dataProvider = \"Options\")\n public void testCompaction(HaloDBOptions options) throws Exception {\n String directory = TestUtils.getTestDirectory(\"HaloDBCompactionTest\", \"testCompaction\");\n\n options.setMaxFileSize(recordsPerFile * recordSize);\n options.setCompactionThresholdPerFile(0.5);\n options.setFlushDataSizeBytes(2048);\n\n HaloDB db = getTestDB(directory, options);\n\n Record[] records = insertAndUpdateRecords(numberOfRecords, db);\n\n TestUtils.waitForCompactionToComplete(db);\n\n for (Record r : records) {\n byte[] actual = db.get(r.getKey());\n Assert.assertEquals(r.getValue(), actual);\n }\n }\n\n @Test(dataProvider = \"Options\")\n public void testReOpenDBAfterCompaction(HaloDBOptions options) throws HaloDBException {\n String directory = TestUtils.getTestDirectory(\"HaloDBCompactionTest\", \"testReOpenDBAfterCompaction\");\n\n options.setMaxFileSize(recordsPerFile * recordSize);\n options.setCompactionThresholdPerFile(0.5);\n\n HaloDB db = getTestDB(directory, options);\n\n Record[] records = insertAndUpdateRecords(numberOfRecords, db);\n\n TestUtils.waitForCompactionToComplete(db);\n\n db.close();\n\n db = getTestDBWithoutDeletingFiles(directory, options);\n\n for (Record r : records) {\n byte[] actual = db.get(r.getKey());\n Assert.assertEquals(actual, r.getValue());\n }\n }\n\n @Test(dataProvider = \"Options\")\n public void testReOpenDBWithoutMerge(HaloDBOptions options) throws HaloDBException {\n String directory = TestUtils.getTestDirectory(\"HaloDBCompactionTest\", \"testReOpenAndUpdatesAndWithoutMerge\");\n\n options.setMaxFileSize(recordsPerFile * recordSize);\n options.setCompactionDisabled(true);\n\n HaloDB db = getTestDB(directory, options);\n\n Record[] records = insertAndUpdateRecords(numberOfRecords, db);\n\n db.close();\n\n db = getTestDBWithoutDeletingFiles(directory, options);\n\n for (Record r : records) {\n byte[] actual = db.get(r.getKey());\n Assert.assertEquals(actual, r.getValue());\n }\n }\n\n @Test(dataProvider = \"Options\")\n public void testSyncWrites(HaloDBOptions options) throws HaloDBException {\n String directory = TestUtils.getTestDirectory(\"HaloDBCompactionTest\", \"testSyncWrites\");\n options.enableSyncWrites(true);\n HaloDB db = getTestDB(directory, options);\n List records = TestUtils.insertRandomRecords(db, 10_000);\n List current = new ArrayList<>();\n for (int i = 0; i < records.size(); i++) {\n if (i % 2 == 0) {\n db.delete(records.get(i).getKey());\n }\n else {\n current.add(records.get(i));\n }\n }\n db.close();\n\n db = getTestDBWithoutDeletingFiles(directory, options);\n Assert.assertEquals(db.size(), current.size());\n for (Record r : current) {\n Assert.assertEquals(db.get(r.getKey()), r.getValue());\n }\n }\n\n @Test(dataProvider = \"Options\")\n public void testUpdatesToSameFile(HaloDBOptions options) throws HaloDBException {\n String directory = TestUtils.getTestDirectory(\"HaloDBCompactionTest\", \"testUpdatesToSameFile\");\n\n options.setMaxFileSize(recordsPerFile * recordSize);\n options.setCompactionDisabled(true);\n\n HaloDB db = getTestDB(directory, options);\n\n Record[] records = insertAndUpdateRecordsToSameFile(2, db);\n\n db.close();\n\n db = getTestDBWithoutDeletingFiles(directory, options);\n\n for (Record r : records) {\n byte[] actual = db.get(r.getKey());\n Assert.assertEquals(actual, r.getValue());\n }\n }\n\n @Test(dataProvider = \"Options\")\n public void testFilesWithStaleDataAddedToCompactionQueueDuringDBOpen(HaloDBOptions options) throws HaloDBException, InterruptedException {\n String directory = TestUtils.getTestDirectory(\"HaloDBCompactionTest\", \"testFilesWithStaleDataAddedToCompactionQueueDuringDBOpen\");\n\n options.setCompactionDisabled(true);\n options.setMaxFileSize(10 * 1024);\n\n HaloDB db = getTestDB(directory, options);\n\n // insert 50 records into 5 files.\n List records = TestUtils.insertRandomRecordsOfSize(db, 50, 1024-Record.Header.HEADER_SIZE);\n\n // Delete all records, which means that all data files would have crossed the\n // stale data threshold. \n for (Record r : records) {\n db.delete(r.getKey());\n }\n\n db.close();\n\n // open the db withe compaction enabled. \n options.setCompactionDisabled(false);\n options.setMaxFileSize(10 * 1024);\n\n db = getTestDBWithoutDeletingFiles(directory, options);\n\n TestUtils.waitForCompactionToComplete(db);\n\n // Since all files have crossed stale data threshold, everything will be compacted and deleted.\n Assert.assertFalse(TestUtils.getLatestDataFile(directory).isPresent());\n Assert.assertFalse(TestUtils.getLatestCompactionFile(directory).isPresent());\n\n db.close();\n\n // open the db with compaction disabled.\n options.setMaxFileSize(10 * 1024);\n options.setCompactionDisabled(true);\n\n db = getTestDBWithoutDeletingFiles(directory, options);\n\n // insert 20 records into two files. \n records = TestUtils.insertRandomRecordsOfSize(db, 20, 1024-Record.Header.HEADER_SIZE);\n File[] dataFilesToDelete = FileUtils.listDataFiles(new File(directory));\n\n // update all records; since compaction is disabled no file is deleted.\n List updatedRecords = TestUtils.updateRecords(db, records);\n\n db.close();\n\n // Open db again with compaction enabled.\n options.setCompactionDisabled(false);\n options.setMaxFileSize(10 * 1024);\n\n db = getTestDBWithoutDeletingFiles(directory, options);\n TestUtils.waitForCompactionToComplete(db);\n\n //Confirm that previous data files were compacted and deleted.\n for (File f : dataFilesToDelete) {\n Assert.assertFalse(f.exists());\n }\n\n for (Record r : updatedRecords) {\n Assert.assertEquals(db.get(r.getKey()), r.getValue());\n }\n }\n\n @Test\n public void testPauseAndResumeCompaction() throws HaloDBException, InterruptedException {\n String directory = TestUtils.getTestDirectory(\"HaloDBCompactionTest\", \"testPauseAndResumeCompaction\");\n\n HaloDBOptions options = new HaloDBOptions();\n options.setMaxFileSize(10 * 1024);\n\n // start compaction immediately after a record is updated.\n options.setCompactionThresholdPerFile(.001);\n\n HaloDB db = getTestDB(directory, options);\n\n // insert 100 records of size 1kb into 100 files.\n int noOfRecords = 1000;\n List records = TestUtils.insertRandomRecordsOfSize(db, noOfRecords, 1024 - Record.Header.HEADER_SIZE);\n List dataFiles = TestUtils.getDataFiles(directory);\n db.pauseCompaction();\n\n // update first record of each file.\n List recordsToUpdate = IntStream.range(0, records.size()).filter(i -> i%10 == 0)\n .mapToObj(i -> records.get(i)).collect(Collectors.toList());\n TestUtils.updateRecordsWithSize(db, recordsToUpdate, 1024-Record.Header.HEADER_SIZE);\n TestUtils.waitForCompactionToComplete(db);\n\n // compaction was paused, therefore no compaction files must be present.\n Assert.assertFalse(TestUtils.getLatestCompactionFile(directory).isPresent());\n\n // resume and pause compaction a few times.\n // each is also called multiple times; duplicate calls shouldn't have any effect.\n db.resumeCompaction();\n Assert.assertTrue(db.stats().isCompactionRunning());\n\n Thread.sleep(5);\n db.pauseCompaction();\n db.pauseCompaction();\n Assert.assertFalse(db.stats().isCompactionRunning());\n TestUtils.waitForCompactionToComplete(db);\n\n Thread.sleep(100);\n db.resumeCompaction();\n db.resumeCompaction();\n Assert.assertTrue(db.stats().isCompactionRunning());\n\n Thread.sleep(20);\n db.pauseCompaction();\n db.pauseCompaction();\n db.pauseCompaction();\n Assert.assertFalse(db.stats().isCompactionRunning());\n TestUtils.waitForCompactionToComplete(db);\n\n Thread.sleep(100);\n db.resumeCompaction();\n db.resumeCompaction();\n Assert.assertTrue(db.stats().isCompactionRunning());\n TestUtils.waitForCompactionToComplete(db);\n\n // compaction files are present.\n Assert.assertTrue(TestUtils.getLatestCompactionFile(directory).isPresent());\n\n // all the data files created before update were deleted by compaction thread.\n dataFiles.forEach(f -> Assert.assertFalse(f.exists(), \"data file \" + f.getName() + \" still exists\"));\n }\n\n private Record[] insertAndUpdateRecords(int numberOfRecords, HaloDB db) throws HaloDBException {\n int valueSize = recordSize - Record.Header.HEADER_SIZE - 8; // 8 is the key size.\n\n Record[] records = new Record[numberOfRecords];\n for (int i = 0; i < numberOfRecords; i++) {\n byte[] key = Longs.toByteArray(i);\n byte[] value = TestUtils.generateRandomByteArray(valueSize);\n records[i] = new Record(key, value);\n db.put(records[i].getKey(), records[i].getValue());\n }\n\n // modify first 5 records of each file.\n byte[] modifiedMark = \"modified\".getBytes();\n for (int k = 0; k < numberOfFiles; k++) {\n for (int i = 0; i < 5; i++) {\n Record r = records[i + k*10];\n byte[] value = r.getValue();\n System.arraycopy(modifiedMark, 0, value, 0, modifiedMark.length);\n Record modifiedRecord = new Record(r.getKey(), value);\n records[i + k*10] = modifiedRecord;\n db.put(modifiedRecord.getKey(), modifiedRecord.getValue());\n }\n }\n return records;\n }\n\n private Record[] insertAndUpdateRecordsToSameFile(int numberOfRecords, HaloDB db) throws HaloDBException {\n int valueSize = recordSize - Record.Header.HEADER_SIZE - 8; // 8 is the key size.\n\n Record[] records = new Record[numberOfRecords];\n for (int i = 0; i < numberOfRecords; i++) {\n byte[] key = Longs.toByteArray(i);\n byte[] value = TestUtils.generateRandomByteArray(valueSize);\n\n byte[] updatedValue = null;\n for (long j = 0; j < recordsPerFile; j++) {\n updatedValue = TestUtils.concatenateArrays(value, Longs.toByteArray(i));\n db.put(key, updatedValue);\n }\n\n // only store the last updated valued.\n records[i] = new Record(key, updatedValue);\n }\n\n return records;\n }\n}\n" }, { "path": "src/test/java/com/oath/halodb/HaloDBDeletionTest.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport org.testng.Assert;\nimport org.testng.annotations.Test;\n\nimport java.io.File;\nimport java.util.ArrayList;\nimport java.util.HashSet;\nimport java.util.List;\nimport java.util.Random;\nimport java.util.Set;\n\npublic class HaloDBDeletionTest extends TestBase {\n\n @Test(dataProvider = \"Options\")\n public void testSimpleDelete(HaloDBOptions options) throws HaloDBException {\n String directory = TestUtils.getTestDirectory(\"HaloDBDeletionTest\", \"testSimpleDelete\");\n options.setCompactionDisabled(true);\n\n HaloDB db = getTestDB(directory, options);\n\n int noOfRecords = 10_000;\n List records = TestUtils.insertRandomRecords(db, noOfRecords);\n\n // delete every other record.\n for (int i = 0; i < records.size(); i++) {\n if (i % 2 == 0) {\n db.delete(records.get(i).getKey());\n }\n }\n\n for (int i = 0; i < records.size(); i++) {\n byte[] actual = db.get(records.get(i).getKey());\n\n if (i % 2 == 0) {\n Assert.assertNull(actual);\n }\n else {\n Assert.assertEquals(records.get(i).getValue(), actual);\n }\n }\n }\n\n @Test(dataProvider = \"Options\")\n public void testDeleteWithIterator(HaloDBOptions options) throws HaloDBException {\n String directory = TestUtils.getTestDirectory(\"HaloDBDeletionTest\", \"testDeleteWithIterator\");\n options.setCompactionDisabled(true);\n\n HaloDB db = getTestDB(directory, options);\n\n int noOfRecords = 10_000;\n List records = TestUtils.insertRandomRecords(db, noOfRecords);\n\n // delete every other record.\n List expected = new ArrayList<>();\n for (int i = 0; i < records.size(); i++) {\n if (i % 2 == 0) {\n db.delete(records.get(i).getKey());\n }\n else {\n expected.add(records.get(i));\n }\n }\n\n List actual = new ArrayList<>();\n db.newIterator().forEachRemaining(actual::add);\n\n Assert.assertTrue(actual.containsAll(expected) && expected.containsAll(actual));\n }\n\n @Test(dataProvider = \"Options\")\n public void testDeleteAndInsert(HaloDBOptions options) throws HaloDBException {\n String directory = TestUtils.getTestDirectory(\"HaloDBDeletionTest\", \"testDeleteAndInsert\");\n options.setCompactionDisabled(true);\n\n HaloDB db = getTestDB(directory, options);\n\n int noOfRecords = 100;\n List records = TestUtils.insertRandomRecords(db, noOfRecords);\n\n // delete every other record.\n for (int i = 0; i < records.size(); i++) {\n if (i % 2 == 0) {\n db.delete(records.get(i).getKey());\n }\n }\n\n for (int i = 0; i < records.size(); i++) {\n byte[] actual = db.get(records.get(i).getKey());\n\n if (i % 2 == 0) {\n Assert.assertNull(actual);\n }\n else {\n Assert.assertEquals(records.get(i).getValue(), actual);\n }\n }\n\n // insert deleted records.\n for (int i = 0; i < records.size(); i++) {\n if (i % 2 == 0) {\n byte[] value = TestUtils.generateRandomByteArray();\n byte[] key = records.get(i).getKey();\n db.put(key, value);\n records.set(i, new Record(key, value));\n }\n }\n\n records.forEach(record -> {\n try {\n byte[] value = db.get(record.getKey());\n Assert.assertEquals(record.getValue(), value);\n } catch (HaloDBException e) {\n throw new RuntimeException(e);\n }\n });\n\n // also check the iterator.\n List actual = new ArrayList<>();\n db.newIterator().forEachRemaining(actual::add);\n\n Assert.assertTrue(actual.containsAll(records) && records.containsAll(actual));\n }\n\n @Test(dataProvider = \"Options\")\n public void testDeleteAndOpen(HaloDBOptions options) throws HaloDBException {\n String directory = TestUtils.getTestDirectory(\"HaloDBDeletionTest\", \"testDeleteAndOpen\");\n options.setCompactionDisabled(true);\n\n HaloDB db = getTestDB(directory, options);\n\n int noOfRecords = 10_000;\n List records = TestUtils.insertRandomRecords(db, noOfRecords);\n\n // delete every other record.\n for (int i = 0; i < records.size(); i++) {\n if (i % 2 == 0) {\n db.delete(records.get(i).getKey());\n }\n }\n\n db.close();\n\n db = getTestDBWithoutDeletingFiles(directory, options);\n\n for (int i = 0; i < records.size(); i++) {\n byte[] actual = db.get(records.get(i).getKey());\n\n if (i % 2 == 0) {\n Assert.assertNull(actual);\n }\n else {\n Assert.assertEquals(records.get(i).getValue(), actual);\n }\n }\n }\n\n @Test(dataProvider = \"Options\")\n public void testDeleteAndMerge(HaloDBOptions options) throws Exception {\n String directory = TestUtils.getTestDirectory(\"HaloDBDeletionTest\", \"testDeleteAndMerge\");\n options.setMaxFileSize(10 * 1024);\n options.setCompactionThresholdPerFile(0.10);\n\n HaloDB db = getTestDB(directory, options);\n\n int noOfRecords = 10_000;\n List records = TestUtils.insertRandomRecords(db, noOfRecords);\n\n // delete records\n Random random = new Random();\n Set deleted = new HashSet<>();\n List newRecords = new ArrayList<>();\n for (int i = 0; i < 1000; i++) {\n int index = random.nextInt(records.size());\n db.delete(records.get(index).getKey());\n deleted.add(index);\n\n // also throw in some new records into to mix.\n // size is 40 so that we create keys distinct from\n // what we used before.\n byte[] key = TestUtils.generateRandomByteArray(40);\n db.put(key, TestUtils.generateRandomByteArray());\n newRecords.add(key);\n }\n\n // update the new records to make sure the the files containing tombstones\n // will be compacted.\n for (byte[] key : newRecords) {\n db.put(key, TestUtils.generateRandomByteArray());\n }\n\n TestUtils.waitForCompactionToComplete(db);\n\n db.close();\n\n db = getTestDBWithoutDeletingFiles(directory, options);\n\n for (int i = 0; i < records.size(); i++) {\n byte[] actual = db.get(records.get(i).getKey());\n\n if (deleted.contains(i)) {\n Assert.assertNull(actual);\n }\n else {\n Assert.assertEquals(records.get(i).getValue(), actual);\n }\n }\n }\n\n @Test(dataProvider = \"Options\")\n public void testDeleteAllRecords(HaloDBOptions options) throws Exception {\n String directory = TestUtils.getTestDirectory(\"HaloDBDeletionTest\", \"testDeleteAllRecords\");\n options.setMaxFileSize(10 * 1024);\n options.setCompactionThresholdPerFile(1);\n\n HaloDB db = getTestDB(directory, options);\n\n int noOfRecords = 10_000;\n // There will be 1000 files each of size 10KB\n List records = TestUtils.insertRandomRecordsOfSize(db, noOfRecords, 1024 - Record.Header.HEADER_SIZE);\n\n // delete all records.\n for (Record r : records) {\n db.delete(r.getKey());\n }\n\n TestUtils.waitForCompactionToComplete(db);\n\n Assert.assertEquals(db.size(), 0);\n\n for (Record r : records) {\n Assert.assertNull(db.get(r.getKey()));\n }\n\n // only the current write file will be remaining everything else should have been\n // deleted by the compaction job. \n Assert.assertEquals(FileUtils.listDataFiles(new File(directory)).length, 1);\n }\n}\n" }, { "path": "src/test/java/com/oath/halodb/HaloDBFileCompactionTest.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport com.google.common.primitives.Longs;\n\nimport org.hamcrest.MatcherAssert;\nimport org.hamcrest.Matchers;\nimport org.testng.annotations.Test;\n\nimport java.io.File;\nimport java.util.ArrayList;\nimport java.util.Arrays;\nimport java.util.Comparator;\nimport java.util.List;\n\npublic class HaloDBFileCompactionTest extends TestBase {\n\n @Test(dataProvider = \"Options\")\n public void testCompaction(HaloDBOptions options) throws Exception {\n String directory = TestUtils.getTestDirectory(\"HaloDBFileCompactionTest\", \"testCompaction\");\n\n int recordSize = 1024;\n int recordNumber = 20;\n\n options.setMaxFileSize(10 * recordSize); // 10 records per data file.\n options.setCompactionThresholdPerFile(0.5);\n\n HaloDB db = getTestDB(directory, options);\n\n byte[] data = new byte[recordSize - Record.Header.HEADER_SIZE - 8 - 8];\n for (int i = 0; i < data.length; i++) {\n data[i] = (byte)i;\n }\n\n Record[] records = new Record[recordNumber];\n for (int i = 0; i < recordNumber; i++) {\n byte[] key = Longs.toByteArray(i);\n byte[] value = TestUtils.concatenateArrays(data, key);\n records[i] = new Record(key, value);\n db.put(records[i].getKey(), records[i].getValue());\n }\n\n List freshRecords = new ArrayList<>();\n\n // There are two data files. make the first half of both the files stale. \n for (int i = 0; i < 5; i++) {\n db.put(records[i].getKey(), records[i].getValue());\n db.put(records[i+10].getKey(), records[i+10].getValue());\n }\n\n // Second half of both the files should be copied to the compacted file.\n for (int i = 5; i < 10; i++) {\n freshRecords.add(records[i]);\n freshRecords.add(records[i + 10]);\n }\n\n TestUtils.waitForCompactionToComplete(db);\n\n // the latest file will be the compacted file.\n File compactedFile = Arrays.stream(FileUtils.listDataFiles(new File(directory))).max(Comparator.comparing(File::getName)).get();\n HaloDBFile.HaloDBFileIterator iterator = HaloDBFile.openForReading(dbDirectory, compactedFile, HaloDBFile.FileType.COMPACTED_FILE, options).newIterator();\n\n // make sure the the compacted file has the bottom half of two files.\n List mergedRecords = new ArrayList<>();\n while (iterator.hasNext()) {\n mergedRecords.add(iterator.next());\n }\n\n MatcherAssert.assertThat(freshRecords, Matchers.containsInAnyOrder(mergedRecords.toArray()));\n }\n}\n" }, { "path": "src/test/java/com/oath/halodb/HaloDBFileTest.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport org.testng.Assert;\nimport org.testng.annotations.AfterMethod;\nimport org.testng.annotations.BeforeMethod;\nimport org.testng.annotations.Test;\n\nimport java.io.File;\nimport java.io.IOException;\nimport java.nio.ByteBuffer;\nimport java.nio.channels.FileChannel;\nimport java.nio.file.Paths;\nimport java.nio.file.StandardOpenOption;\nimport java.nio.file.attribute.FileTime;\nimport java.util.List;\n\npublic class HaloDBFileTest {\n\n private File directory = Paths.get(\"tmp\", \"HaloDBFileTest\", \"testIndexFile\").toFile();\n private DBDirectory dbDirectory;\n private HaloDBFile file;\n private IndexFile indexFile;\n private int fileId = 100;\n private File backingFile = directory.toPath().resolve(fileId+HaloDBFile.DATA_FILE_NAME).toFile();\n private FileTime createdTime;\n\n @BeforeMethod\n public void before() throws IOException {\n TestUtils.deleteDirectory(directory);\n dbDirectory = DBDirectory.open(directory);\n file = HaloDBFile.create(dbDirectory, fileId, new HaloDBOptions(), HaloDBFile.FileType.DATA_FILE);\n createdTime = TestUtils.getFileCreationTime(backingFile);\n indexFile = new IndexFile(fileId, dbDirectory, new HaloDBOptions());\n try {\n // wait for a second to make sure that the file creation time of the repaired file will be different.\n Thread.sleep(1000);\n } catch (InterruptedException e) {\n }\n }\n\n @AfterMethod\n public void after() throws IOException {\n if (file != null)\n file.close();\n if (indexFile != null)\n indexFile.close();\n dbDirectory.close();\n TestUtils.deleteDirectory(directory);\n }\n\n @Test\n public void testIndexFile() throws IOException {\n List list = insertTestRecords();\n\n indexFile.open();\n verifyIndexFile(indexFile, list);\n }\n\n @Test\n public void testFileWithInvalidRecord() throws IOException {\n List list = insertTestRecords();\n\n // write a corrupted header to file.\n try(FileChannel channel = FileChannel.open(Paths.get(directory.getCanonicalPath(), fileId + HaloDBFile.DATA_FILE_NAME).toAbsolutePath(), StandardOpenOption.APPEND)) {\n ByteBuffer data = ByteBuffer.wrap(\"garbage\".getBytes());\n channel.write(data);\n }\n\n HaloDBFile.HaloDBFileIterator iterator = file.newIterator();\n int count = 0;\n while (iterator.hasNext() && count < 100) {\n Record record = iterator.next();\n Assert.assertEquals(record.getKey(), list.get(count++).getKey());\n }\n\n // 101th record's header is corrupted.\n Assert.assertTrue(iterator.hasNext());\n // Since header is corrupted we won't be able to read it and hence next will return null. \n Assert.assertNull(iterator.next());\n }\n\n @Test\n public void testCorruptedHeader() throws IOException {\n List list = insertTestRecords();\n\n // write a corrupted header to file.\n // write a corrupted record to file.\n byte[] key = \"corrupted key\".getBytes();\n byte[] value = \"corrupted value\".getBytes();\n Record corrupted = new Record(key, value);\n // value length is corrupted. \n corrupted.setHeader(new Record.Header(0, 0, (byte)key.length, -345445, 1234));\n try(FileChannel channel = FileChannel.open(Paths.get(directory.getCanonicalPath(), fileId + HaloDBFile.DATA_FILE_NAME).toAbsolutePath(), StandardOpenOption.APPEND)) {\n channel.write(corrupted.serialize());\n }\n\n HaloDBFile.HaloDBFileIterator iterator = file.newIterator();\n int count = 0;\n while (iterator.hasNext() && count < 100) {\n Record r = iterator.next();\n Assert.assertEquals(r.getKey(), list.get(count).getKey());\n Assert.assertEquals(r.getValue(), list.get(count).getValue());\n count++;\n }\n\n // 101th record's header is corrupted.\n Assert.assertTrue(iterator.hasNext());\n // Since header is corrupted we won't be able to read it and hence next will return null.\n Assert.assertNull(iterator.next());\n }\n\n @Test\n public void testRebuildIndexFile() throws IOException {\n List list = insertTestRecords();\n\n indexFile.delete();\n\n // make sure that the file is deleted. \n Assert.assertFalse(Paths.get(directory.getName(), fileId + IndexFile.INDEX_FILE_NAME).toFile().exists());\n file.rebuildIndexFile();\n indexFile.open();\n verifyIndexFile(indexFile, list);\n }\n\n @Test\n public void testRepairDataFileWithCorruptedValue() throws IOException {\n List list = insertTestRecords();\n\n // write a corrupted record to file.\n // the record is corrupted in such a way the the size is unchanged but the contents have changed, thus crc will be different. \n byte[] key = \"corrupted key\".getBytes();\n byte[] value = \"corrupted value\".getBytes();\n Record record = new Record(key, value);\n record.setHeader(new Record.Header(0, 2, (byte)key.length, value.length, 1234));\n try(FileChannel channel = FileChannel.open(Paths.get(directory.getCanonicalPath(), fileId + HaloDBFile.DATA_FILE_NAME).toAbsolutePath(), StandardOpenOption.APPEND)) {\n ByteBuffer[] data = record.serialize();\n data[2] = ByteBuffer.wrap(\"value corrupted\".getBytes());\n channel.write(data);\n }\n\n HaloDBFile repairedFile = file.repairFile(dbDirectory);\n Assert.assertNotEquals(TestUtils.getFileCreationTime(backingFile), createdTime);\n Assert.assertEquals(repairedFile.getPath(), file.getPath());\n verifyDataFile(list, repairedFile);\n verifyIndexFile(repairedFile.getIndexFile(), list);\n }\n\n @Test\n public void testRepairDataFileWithInCompleteRecord() throws IOException {\n List list = insertTestRecords();\n\n // write a corrupted record to file.\n // value was not completely written to file. \n byte[] key = \"corrupted key\".getBytes();\n byte[] value = \"corrupted value\".getBytes();\n Record record = new Record(key, value);\n record.setHeader(new Record.Header(0, 100, (byte)key.length, value.length, 1234));\n try(FileChannel channel = FileChannel.open(Paths.get(directory.getCanonicalPath(), fileId + HaloDBFile.DATA_FILE_NAME).toAbsolutePath(), StandardOpenOption.APPEND)) {\n ByteBuffer[] data = record.serialize();\n data[2] = ByteBuffer.wrap(\"missing\".getBytes());\n channel.write(data);\n }\n\n HaloDBFile repairedFile = file.repairFile(dbDirectory);\n Assert.assertNotEquals(TestUtils.getFileCreationTime(backingFile), createdTime);\n Assert.assertEquals(repairedFile.getPath(), file.getPath());\n verifyDataFile(list, repairedFile);\n verifyIndexFile(repairedFile.getIndexFile(), list);\n }\n\n @Test\n public void testRepairDataFileContainingRecordsWithCorruptedHeader() throws IOException {\n List list = insertTestRecords();\n\n // write a corrupted header to file.\n try(FileChannel channel = FileChannel.open(Paths.get(directory.getCanonicalPath(), fileId + HaloDBFile.DATA_FILE_NAME).toAbsolutePath(), StandardOpenOption.APPEND)) {\n ByteBuffer data = ByteBuffer.wrap(\"garbage\".getBytes());\n channel.write(data);\n }\n\n HaloDBFile repairedFile = file.repairFile(dbDirectory);\n Assert.assertNotEquals(TestUtils.getFileCreationTime(backingFile), createdTime);\n Assert.assertEquals(repairedFile.getPath(), file.getPath());\n verifyDataFile(list, repairedFile);\n verifyIndexFile(repairedFile.getIndexFile(), list);\n }\n\n @Test\n public void testRepairDataFileContainingRecordsWithValidButCorruptedHeader() throws IOException {\n List list = insertTestRecords();\n\n // write a corrupted record to file.\n byte[] key = \"corrupted key\".getBytes();\n byte[] value = \"corrupted value\".getBytes();\n Record record = new Record(key, value);\n // header is valid but the value size is incorrect. \n record.setHeader(new Record.Header(0,101, (byte)key.length, 5, 1234));\n try(FileChannel channel = FileChannel.open(Paths.get(directory.getCanonicalPath(), fileId + HaloDBFile.DATA_FILE_NAME).toAbsolutePath(), StandardOpenOption.APPEND)) {\n ByteBuffer[] data = record.serialize();\n channel.write(data);\n }\n\n HaloDBFile repairedFile = file.repairFile(dbDirectory);\n Assert.assertNotEquals(TestUtils.getFileCreationTime(backingFile), createdTime);\n Assert.assertEquals(repairedFile.getPath(), file.getPath());\n verifyDataFile(list, repairedFile);\n verifyIndexFile(repairedFile.getIndexFile(), list);\n }\n\n private void verifyIndexFile(IndexFile file, List recordList) throws IOException {\n IndexFile.IndexFileIterator indexFileIterator = file.newIterator();\n int count = 0;\n while (indexFileIterator.hasNext()) {\n IndexFileEntry e = indexFileIterator.next();\n Record r = recordList.get(count++);\n InMemoryIndexMetaData meta = r.getRecordMetaData();\n Assert.assertEquals(e.getKey(), r.getKey());\n\n int expectedOffset = meta.getValueOffset() - Record.Header.HEADER_SIZE - r.getKey().length;\n Assert.assertEquals(e.getRecordOffset(), expectedOffset);\n }\n\n Assert.assertEquals(count, recordList.size());\n }\n\n private List insertTestRecords() throws IOException {\n List list = TestUtils.generateRandomData(100);\n for (Record record : list) {\n record.setSequenceNumber(100);\n InMemoryIndexMetaData meta = file.writeRecord(record);\n record.setRecordMetaData(meta);\n }\n return list;\n }\n\n private void verifyDataFile(List recordList, HaloDBFile dataFile) throws IOException {\n HaloDBFile.HaloDBFileIterator iterator = dataFile.newIterator();\n int count = 0;\n while (iterator.hasNext()) {\n Record actual = iterator.next();\n Record expected = recordList.get(count++);\n Assert.assertEquals(actual, expected);\n }\n\n Assert.assertEquals(count, recordList.size());\n }\n}\n" }, { "path": "src/test/java/com/oath/halodb/HaloDBIteratorTest.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport org.hamcrest.MatcherAssert;\nimport org.hamcrest.Matchers;\nimport org.testng.Assert;\nimport org.testng.annotations.Test;\n\nimport java.io.IOException;\nimport java.nio.channels.ClosedChannelException;\nimport java.util.ArrayList;\nimport java.util.List;\nimport java.util.NoSuchElementException;\n\nimport mockit.Invocation;\nimport mockit.Mock;\nimport mockit.MockUp;\n\npublic class HaloDBIteratorTest extends TestBase {\n\n @Test(expectedExceptions = NoSuchElementException.class, dataProvider = \"Options\")\n public void testWithEmptyDB(HaloDBOptions options) throws HaloDBException {\n String directory = TestUtils.getTestDirectory(\"HaloDBIteratorTest\", \"testWithEmptyDB\");\n\n HaloDB db = getTestDB(directory, options);\n HaloDBIterator iterator = db.newIterator();\n Assert.assertFalse(iterator.hasNext());\n iterator.next();\n }\n\n @Test(dataProvider = \"Options\")\n public void testWithDelete(HaloDBOptions options) throws HaloDBException {\n String directory = TestUtils.getTestDirectory(\"HaloDBIteratorTest\", \"testWithEmptyDB\");\n\n options.setCompactionDisabled(true);\n\n HaloDB db = getTestDB(directory, options);\n int noOfRecords = 10_000;\n List records = TestUtils.insertRandomRecords(db, noOfRecords);\n\n // delete all records.\n for (Record r : records) {\n db.delete(r.getKey());\n }\n\n HaloDBIterator iterator = db.newIterator();\n Assert.assertFalse(iterator.hasNext());\n\n // close and open the db again. \n db.close();\n db = getTestDBWithoutDeletingFiles(directory, options);\n iterator = db.newIterator();\n Assert.assertFalse(iterator.hasNext());\n }\n\n @Test(dataProvider = \"Options\")\n public void testPutAndGetDB(HaloDBOptions options) throws HaloDBException {\n String directory = TestUtils.getTestDirectory(\"HaloDBIteratorTest\", \"testPutAndGetDB\");\n\n options.setCompactionDisabled(true);\n options.setMaxFileSize(10 * 1024);\n\n HaloDB db = getTestDB(directory, options);\n\n int noOfRecords = 10_000;\n List records = TestUtils.insertRandomRecords(db, noOfRecords);\n\n List actual = new ArrayList<>();\n db.newIterator().forEachRemaining(actual::add);\n\n MatcherAssert.assertThat(actual, Matchers.containsInAnyOrder(records.toArray()));\n }\n\n @Test(dataProvider = \"Options\")\n public void testPutUpdateAndGetDB(HaloDBOptions options) throws HaloDBException {\n String directory = TestUtils.getTestDirectory(\"HaloDBIteratorTest\", \"testPutUpdateAndGetDB\");\n\n options.setCompactionDisabled(true);\n options.setMaxFileSize(10 * 1024);\n\n HaloDB db = getTestDB(directory, options);\n\n int noOfRecords = 10_000;\n List records = TestUtils.insertRandomRecords(db, noOfRecords);\n\n List updated = TestUtils.updateRecords(db, records);\n\n List actual = new ArrayList<>();\n db.newIterator().forEachRemaining(actual::add);\n MatcherAssert.assertThat(actual, Matchers.containsInAnyOrder(updated.toArray()));\n }\n\n @Test(dataProvider = \"Options\")\n public void testPutUpdateCompactAndGetDB(HaloDBOptions options) throws HaloDBException, InterruptedException {\n String directory = TestUtils.getTestDirectory(\"HaloDBIteratorTest\", \"testPutUpdateMergeAndGetDB\");\n\n options.setMaxFileSize(10 * 1024);\n options.setCompactionThresholdPerFile(0.50);\n\n HaloDB db = getTestDB(directory, options);\n\n int noOfRecords = 10_000;\n List records = TestUtils.insertRandomRecords(db, noOfRecords);\n\n List updated = TestUtils.updateRecords(db, records);\n\n TestUtils.waitForCompactionToComplete(db);\n\n List actual = new ArrayList<>();\n db.newIterator().forEachRemaining(actual::add);\n\n Assert.assertEquals(actual.size(), updated.size());\n MatcherAssert.assertThat(actual, Matchers.containsInAnyOrder(updated.toArray()));\n }\n\n // Test to make sure that no exceptions are thrown when files are being deleted by\n // compaction thread and db is being iterated. \n @Test(dataProvider = \"Options\")\n public void testConcurrentCompactionAndIterator(HaloDBOptions options) throws HaloDBException, InterruptedException {\n String directory = TestUtils.getTestDirectory(\"HaloDBIteratorTest\", \"testConcurrentCompactionAndIterator\");\n\n options.setMaxFileSize(1024 * 1024);\n options.setCompactionThresholdPerFile(0.1);\n\n final HaloDB db = getTestDB(directory, options);\n\n // insert 1024 records per file, and a total of 10 files.\n int noOfRecords = 10*1024;\n List records = TestUtils.insertRandomRecordsOfSize(db, noOfRecords, 1024-Record.Header.HEADER_SIZE);\n\n int noOfUpdateRuns = 10;\n Thread updateThread = new Thread(() -> {\n for (int i=0; i() {\n\n @Mock\n byte[] readFromFile(Invocation invocation, int offset, int length) throws HaloDBException {\n try {\n // In the iterator after reading from keyCache pause for a while\n // to increase the chance of file being closed by compaction thread.\n Thread.sleep(100);\n } catch (InterruptedException e) {\n }\n\n return invocation.proceed(offset, length);\n }\n\n };\n\n options.setMaxFileSize(2 * 1024);\n options.setCompactionThresholdPerFile(0.1);\n\n final HaloDB db = getTestDB(directory, options);\n\n int noOfRecords = 4; // 2 records on 2 files. \n List records = TestUtils.insertRandomRecordsOfSize(db, noOfRecords, 1024-Record.Header.HEADER_SIZE);\n\n int noOfUpdateRuns = 1000;\n Thread updateThread = new Thread(() -> {\n for (int i=0; i() {\n int count = 0;\n\n @Mock\n byte[] readFromFile(Invocation invocation, int offset, int length) throws IOException {\n if (count % 3 == 0) {\n throw new ClosedChannelException();\n }\n return invocation.proceed(offset, length);\n }\n };\n\n options.setMaxFileSize(10 * 1024);\n options.setCompactionThresholdPerFile(0.6);\n\n final HaloDB db = getTestDB(directory, options);\n\n int noOfRecords = 50; // 50 records on 5 files.\n List records = TestUtils.insertRandomRecordsOfSize(db, noOfRecords, 1024-Record.Header.HEADER_SIZE);\n\n int noOfUpdateRuns = 100;\n Thread updateThread = new Thread(() -> {\n for (int i=0; i records = TestUtils.insertRandomRecords(db, noOfRecords);\n\n // delete all records.\n for (Record r : records) {\n db.delete(r.getKey());\n }\n\n HaloDBKeyIterator iterator = db.newKeyIterator();\n Assert.assertFalse(iterator.hasNext());\n\n // close and open the db again.\n db.close();\n db = getTestDBWithoutDeletingFiles(directory, options);\n iterator = db.newKeyIterator();\n Assert.assertFalse(iterator.hasNext());\n }\n\n @Test(dataProvider = \"Options\")\n public void testPutAndGetDB(HaloDBOptions options) throws HaloDBException {\n String directory = TestUtils.getTestDirectory(\"HaloDBKeyIteratorTest\", \"testPutAndGetDB\");\n\n options.setCompactionDisabled(true);\n options.setMaxFileSize(10 * 1024);\n\n HaloDB db = getTestDB(directory, options);\n\n int noOfRecords = 10_000;\n List records = TestUtils.insertRandomRecords(db, noOfRecords);\n\n List keys = new LinkedList<>();\n for (Record record : records) {\n keys.add(new RecordKey(record.getKey()));\n }\n\n List actual = new ArrayList<>();\n db.newKeyIterator().forEachRemaining(actual::add);\n\n MatcherAssert.assertThat(actual, Matchers.containsInAnyOrder(keys.toArray()));\n }\n\n @Test(dataProvider = \"Options\")\n public void testPutUpdateAndGetDB(HaloDBOptions options) throws HaloDBException {\n String directory = TestUtils.getTestDirectory(\"HaloDBKeyIteratorTest\", \"testPutUpdateAndGetDB\");\n\n options.setCompactionDisabled(true);\n options.setMaxFileSize(10 * 1024);\n\n HaloDB db = getTestDB(directory, options);\n\n int noOfRecords = 10_000;\n List records = TestUtils.insertRandomRecords(db, noOfRecords);\n\n List updated = TestUtils.updateRecords(db, records);\n\n List keys = new LinkedList<>();\n for (Record record : updated) {\n keys.add(new RecordKey(record.getKey()));\n }\n\n List actual = new ArrayList<>();\n db.newKeyIterator().forEachRemaining(actual::add);\n MatcherAssert.assertThat(actual, Matchers.containsInAnyOrder(keys.toArray()));\n }\n}\n" }, { "path": "src/test/java/com/oath/halodb/HaloDBOptionsTest.java", "content": "package com.oath.halodb;\n\nimport org.testng.Assert;\nimport org.testng.annotations.Test;\n\nimport static org.assertj.core.api.Assertions.assertThatIllegalArgumentException;\n\npublic class HaloDBOptionsTest extends TestBase {\n\n @Test\n public void testDefaultOptions() throws HaloDBException {\n String directory = TestUtils.getTestDirectory(\"HaloDBOptionsTest\", \"testDefaultOptions\");\n\n HaloDB db = getTestDB(directory, new HaloDBOptions());\n Assert.assertFalse(db.stats().getOptions().isSyncWrite());\n Assert.assertFalse(db.stats().getOptions().isCompactionDisabled());\n Assert.assertEquals(db.stats().getOptions().getBuildIndexThreads(), 1);\n }\n\n @Test\n public void testSetBuildIndexThreads() {\n int availableProcessors = Runtime.getRuntime().availableProcessors();\n HaloDBOptions options = new HaloDBOptions();\n\n // Test valid boundaries.\n if (availableProcessors > 1) {\n options.setBuildIndexThreads(availableProcessors);\n Assert.assertEquals(options.getBuildIndexThreads(), availableProcessors);\n }\n options.setBuildIndexThreads(1);\n Assert.assertEquals(options.getBuildIndexThreads(), 1);\n\n // Test invalid boundaries.\n assertThatIllegalArgumentException().isThrownBy(() -> options.setBuildIndexThreads(0));\n assertThatIllegalArgumentException().isThrownBy(() -> options.setBuildIndexThreads(availableProcessors + 1));\n }\n}\n" }, { "path": "src/test/java/com/oath/halodb/HaloDBStatsTest.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport org.testng.Assert;\nimport org.testng.annotations.Test;\n\nimport java.util.Arrays;\nimport java.util.List;\nimport java.util.Map;\n\npublic class HaloDBStatsTest extends TestBase {\n\n @Test(dataProvider = \"Options\")\n public void testOptions(HaloDBOptions options) throws HaloDBException {\n String dir = TestUtils.getTestDirectory(\"HaloDBStatsTest\", \"testOptions\");\n\n options.setMaxFileSize(10 * 1024);\n options.setCompactionDisabled(true);\n options.setCompactionThresholdPerFile(0.9);\n options.setFlushDataSizeBytes(1024);\n options.setCompactionJobRate(2048);\n options.setNumberOfRecords(100);\n options.setCleanUpInMemoryIndexOnClose(true);\n\n HaloDB db = getTestDB(dir, options);\n\n HaloDBStats stats = db.stats();\n HaloDBOptions actual = stats.getOptions();\n\n Assert.assertEquals(actual.getMaxFileSize(), options.getMaxFileSize());\n Assert.assertEquals(actual.getCompactionThresholdPerFile(), options.getCompactionThresholdPerFile());\n Assert.assertEquals(actual.getFlushDataSizeBytes(), options.getFlushDataSizeBytes());\n Assert.assertEquals(actual.getCompactionJobRate(), options.getCompactionJobRate());\n Assert.assertEquals(actual.getNumberOfRecords(), options.getNumberOfRecords());\n Assert.assertEquals(actual.isCleanUpInMemoryIndexOnClose(), options.isCleanUpInMemoryIndexOnClose());\n Assert.assertEquals(stats.getSize(), 0);\n }\n\n @Test(dataProvider = \"Options\")\n public void testStaleMap(HaloDBOptions options) throws HaloDBException {\n\n String dir = TestUtils.getTestDirectory(\"HaloDBStatsTest\", \"testStaleMap\");\n\n options.setMaxFileSize(10 * 1024);\n options.setCompactionThresholdPerFile(0.50);\n\n HaloDB db = getTestDB(dir, options);\n\n // will create 10 files with 10 records each. \n int recordSize = 1024 - Record.Header.HEADER_SIZE;\n List records = TestUtils.insertRandomRecordsOfSize(db, 100, recordSize);\n\n // No updates hence stale data map should be empty. \n Assert.assertEquals(db.stats().getStaleDataPercentPerFile().size(), 0);\n\n for (int i = 0; i < records.size(); i++) {\n if (i % 10 == 0)\n db.put(records.get(i).getKey(), TestUtils.generateRandomByteArray(recordSize));\n }\n\n // Updated 1 out of 10 records in each file, hence 10% stale data. \n Assert.assertEquals(db.stats().getStaleDataPercentPerFile().size(), 10);\n db.stats().getStaleDataPercentPerFile().forEach((k, v) -> {\n Assert.assertEquals(v, 10.0);\n });\n\n Assert.assertEquals(db.stats().getSize(), 100);\n }\n\n @Test(dataProvider = \"Options\")\n public void testCompactionStats(HaloDBOptions options) throws HaloDBException {\n\n String dir = TestUtils.getTestDirectory(\"HaloDBStatsTest\", \"testCompactionStats\");\n\n options.setMaxFileSize(10 * 1024);\n options.setCompactionThresholdPerFile(0.50);\n options.setCompactionDisabled(true);\n\n HaloDB db = getTestDB(dir, options);\n // will create 10 files with 10 records each.\n int recordSize = 1024 - Record.Header.HEADER_SIZE;\n int noOfRecords = 100;\n List records = TestUtils.insertRandomRecordsOfSize(db, noOfRecords, recordSize);\n\n Assert.assertEquals(db.stats().getNumberOfDataFiles(), 10);\n Assert.assertEquals(db.stats().getNumberOfTombstoneFiles(), 0);\n\n // update 50% of records in each file. \n for (int i = 0; i < records.size(); i++) {\n if (i % 10 < 5)\n db.put(records.get(i).getKey(), TestUtils.generateRandomByteArray(records.get(i).getValue().length));\n }\n\n TestUtils.waitForCompactionToComplete(db);\n\n // compaction stats are 0 since compaction is paused.\n Assert.assertFalse(db.stats().isCompactionRunning());\n Assert.assertEquals(db.stats().getCompactionRateInInternal(), 0);\n Assert.assertEquals(db.stats().getCompactionRateSinceBeginning(), 0);\n Assert.assertNotEquals(db.stats().toString().length(), 0);\n Assert.assertEquals(db.stats().getNumberOfDataFiles(), 15);\n Assert.assertEquals(db.stats().getNumberOfTombstoneFiles(), 0);\n\n db.close();\n\n options.setCompactionDisabled(false);\n db = getTestDBWithoutDeletingFiles(dir, options);\n Assert.assertTrue(db.stats().isCompactionRunning());\n\n TestUtils.waitForCompactionToComplete(db);\n\n // compaction complete hence stale data map is empty. \n HaloDBStats stats = db.stats();\n Assert.assertEquals(stats.getStaleDataPercentPerFile().size(), 0);\n\n Assert.assertEquals(stats.getNumberOfFilesPendingCompaction(), 0);\n Assert.assertEquals(stats.getNumberOfRecordsCopied(), noOfRecords / 2);\n Assert.assertEquals(stats.getNumberOfRecordsReplaced(), noOfRecords / 2);\n Assert.assertEquals(stats.getNumberOfRecordsScanned(), noOfRecords);\n Assert.assertEquals(stats.getSizeOfRecordsCopied(), noOfRecords / 2 * 1024);\n Assert.assertEquals(stats.getSizeOfFilesDeleted(), options.getMaxFileSize() * 10);\n Assert.assertEquals(stats.getSizeReclaimed(), options.getMaxFileSize() * 10 / 2);\n Assert.assertEquals(stats.getSize(), noOfRecords);\n Assert.assertNotEquals(db.stats().getCompactionRateInInternal(), 0);\n Assert.assertNotEquals(db.stats().getCompactionRateSinceBeginning(), 0);\n Assert.assertNotEquals(db.stats().toString().length(), 0);\n Assert.assertEquals(db.stats().getNumberOfDataFiles(), 10);\n Assert.assertEquals(db.stats().getNumberOfTombstoneFiles(), 0);\n\n // delete the 50% of records\n for (int i = 0; i < records.size(); i++) {\n if (i % 10 < 5)\n db.delete(records.get(i).getKey());\n }\n\n // restart compaction thread because it was stopped by TestUtils.waitForCompactionToComplete(db)\n db.resumeCompaction();\n\n TestUtils.waitForCompactionToComplete(db);\n Assert.assertEquals(db.stats().getNumberOfDataFiles(), 5);\n Assert.assertEquals(db.stats().getNumberOfTombstoneFiles(), 1);\n\n db.close();\n\n // all delete records have been removed by compaction job, so no record copied at reopen\n options.setCleanUpTombstonesDuringOpen(true);\n db = getTestDBWithoutDeletingFiles(dir, options);\n stats = db.stats();\n Assert.assertEquals(stats.getSize(), noOfRecords / 2);\n Assert.assertEquals(db.stats().getNumberOfTombstonesCleanedUpDuringOpen(), noOfRecords / 2);\n Assert.assertEquals(db.stats().getNumberOfTombstonesFoundDuringOpen(), noOfRecords / 2);\n Assert.assertEquals(db.stats().getNumberOfDataFiles(), 5);\n Assert.assertEquals(db.stats().getNumberOfTombstoneFiles(), 0);\n\n db.resetStats();\n stats = db.stats();\n Assert.assertEquals(stats.getNumberOfFilesPendingCompaction(), 0);\n Assert.assertEquals(stats.getNumberOfRecordsCopied(), 0);\n Assert.assertEquals(stats.getNumberOfRecordsReplaced(), 0);\n Assert.assertEquals(stats.getNumberOfRecordsScanned(), 0);\n Assert.assertEquals(stats.getSizeOfRecordsCopied(), 0);\n Assert.assertEquals(stats.getSizeOfFilesDeleted(), 0);\n Assert.assertEquals(stats.getSizeReclaimed(), 0);\n Assert.assertEquals(stats.getSize(), noOfRecords / 2);\n Assert.assertEquals(db.stats().getCompactionRateInInternal(), 0);\n Assert.assertNotEquals(db.stats().getCompactionRateSinceBeginning(), 0);\n Assert.assertNotEquals(db.stats().toString().length(), 0);\n }\n\n @Test(dataProvider = \"Options\")\n public void testIndexStats(HaloDBOptions options) throws HaloDBException {\n String dir = TestUtils.getTestDirectory(\"HaloDBStatsTest\", \"testIndexStats\");\n\n int numberOfSegments = (int)Utils.roundUpToPowerOf2(Runtime.getRuntime().availableProcessors() * 2);\n int numberOfRecords = numberOfSegments * 1024;\n\n options.setMaxFileSize(10 * 1024);\n options.setNumberOfRecords(numberOfRecords);\n options.setCompactionThresholdPerFile(0.50);\n options.setCompactionDisabled(true);\n\n HaloDB db = getTestDB(dir, options);\n HaloDBStats stats = db.stats();\n Assert.assertEquals(numberOfSegments, stats.getNumberOfSegments());\n Assert.assertEquals(numberOfRecords/numberOfSegments, stats.getMaxSizePerSegment());\n\n SegmentStats[] expected = new SegmentStats[numberOfSegments];\n SegmentStats s;\n if (options.isUseMemoryPool()) {\n s = new SegmentStats(0, 0, 0, 0);\n }\n else {\n s = new SegmentStats(0, -1, -1, -1);\n }\n\n Arrays.fill(expected, s);\n Assert.assertEquals(stats.getSegmentStats(), expected);\n }\n\n @Test(dataProvider = \"Options\")\n public void testStatsToStringMap(HaloDBOptions options) throws HaloDBException {\n String dir = TestUtils.getTestDirectory(\"HaloDBStatsTest\", \"testStatsToStringMap\");\n\n HaloDB db = getTestDB(dir, options);\n\n HaloDBStats stats = db.stats();\n Map map = stats.toStringMap();\n Assert.assertEquals(map.size(), 22);\n Assert.assertNotNull(map.get(\"statsResetTime\"));\n Assert.assertNotNull(map.get(\"size\"));\n Assert.assertNotNull(map.get(\"Options\"));\n Assert.assertNotNull(map.get(\"isCompactionRunning\"));\n Assert.assertNotNull(map.get(\"CompactionJobRateInInterval\"));\n Assert.assertNotNull(map.get(\"CompactionJobRateSinceBeginning\"));\n Assert.assertNotNull(map.get(\"numberOfFilesPendingCompaction\"));\n Assert.assertNotNull(map.get(\"numberOfRecordsCopied\"));\n Assert.assertNotNull(map.get(\"numberOfRecordsReplaced\"));\n Assert.assertNotNull(map.get(\"numberOfRecordsScanned\"));\n Assert.assertNotNull(map.get(\"sizeOfRecordsCopied\"));\n Assert.assertNotNull(map.get(\"sizeOfFilesDeleted\"));\n Assert.assertNotNull(map.get(\"sizeReclaimed\"));\n Assert.assertNotNull(map.get(\"rehashCount\"));\n Assert.assertNotNull(map.get(\"maxSizePerSegment\"));\n Assert.assertNotNull(map.get(\"numberOfDataFiles\"));\n Assert.assertNotNull(map.get(\"numberOfTombstoneFiles\"));\n Assert.assertNotNull(map.get(\"numberOfTombstonesFoundDuringOpen\"));\n Assert.assertNotNull(map.get(\"numberOfTombstonesCleanedUpDuringOpen\"));\n Assert.assertNotNull(map.get(\"segmentStats\"));\n Assert.assertNotNull(map.get(\"numberOfSegments\"));\n Assert.assertNotNull(map.get(\"staleDataPercentPerFile\"));\n }\n\n}\n" }, { "path": "src/test/java/com/oath/halodb/HaloDBTest.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport com.google.common.primitives.Longs;\n\nimport java.io.File;\nimport java.nio.ByteBuffer;\nimport java.util.*;\nimport org.testng.Assert;\nimport org.testng.annotations.Test;\n\nimport java.io.IOException;\nimport java.nio.file.Paths;\n\nimport mockit.Mock;\nimport mockit.MockUp;\n\npublic class HaloDBTest extends TestBase {\n\n @Test(dataProvider = \"Options\")\n public void testPutAndGetDB(HaloDBOptions options) throws HaloDBException {\n String directory = TestUtils.getTestDirectory(\"HaloDBTest\", \"testPutAndGetDB\");\n\n options.setCompactionDisabled(true);\n\n HaloDB db = getTestDB(directory, options);\n\n int noOfRecords = 10_000;\n List records = TestUtils.insertRandomRecords(db, noOfRecords);\n\n List actual = new ArrayList<>();\n db.newIterator().forEachRemaining(actual::add);\n\n Assert.assertTrue(actual.containsAll(records) && records.containsAll(actual));\n\n records.forEach(record -> {\n try {\n byte[] value = db.get(record.getKey());\n Assert.assertEquals(record.getValue(), value);\n } catch (HaloDBException e) {\n throw new RuntimeException(e);\n }\n });\n }\n\n @Test(dataProvider = \"Options\")\n public void testPutUpdateAndGetDB(HaloDBOptions options) throws HaloDBException {\n String directory = TestUtils.getTestDirectory(\"HaloDBTest\", \"testPutUpdateAndGetDB\");\n\n options.setCompactionDisabled(true);\n options.setMaxFileSize(10 * 1024);\n\n HaloDB db = getTestDB(directory, options);\n\n int noOfRecords = 10_000;\n List records = TestUtils.insertRandomRecords(db, noOfRecords);\n\n List updated = TestUtils.updateRecords(db, records);\n\n List actual = new ArrayList<>();\n db.newIterator().forEachRemaining(actual::add);\n\n Assert.assertTrue(actual.containsAll(updated) && updated.containsAll(actual));\n\n updated.forEach(record -> {\n try {\n byte[] value = db.get(record.getKey());\n Assert.assertEquals(record.getValue(), value);\n } catch (HaloDBException e) {\n throw new RuntimeException(e);\n }\n });\n }\n\n @Test(dataProvider = \"Options\")\n public void testCreateCloseAndOpenDB(HaloDBOptions options) throws HaloDBException {\n\n String directory = TestUtils.getTestDirectory(\"HaloDBTest\", \"testCreateCloseAndOpenDB\");\n\n options.setCompactionDisabled(true);\n options.setMaxFileSize(10 * 1024);\n\n HaloDB db = getTestDB(directory, options);\n\n int noOfRecords = 10_000;\n List records = TestUtils.insertRandomRecords(db, noOfRecords);\n\n // update half the records.\n for (int i = 0; i < records.size(); i++) {\n if (i % 2 == 0) {\n Record record = records.get(i);\n try {\n byte[] value = TestUtils.generateRandomByteArray();\n db.put(record.getKey(), value);\n records.set(i, new Record(record.getKey(), value));\n } catch (HaloDBException e) {\n throw new RuntimeException(e);\n }\n }\n\n }\n\n db.close();\n\n // open and read contents again.\n HaloDB openAgainDB = getTestDBWithoutDeletingFiles(directory, options);\n\n List actual = new ArrayList<>();\n openAgainDB.newIterator().forEachRemaining(actual::add);\n\n Assert.assertTrue(actual.containsAll(records) && records.containsAll(actual));\n\n records.forEach(record -> {\n try {\n byte[] value = openAgainDB.get(record.getKey());\n Assert.assertEquals(record.getValue(), value);\n } catch (HaloDBException e) {\n throw new RuntimeException(e);\n }\n });\n }\n\n @Test(dataProvider = \"Options\")\n public void testSyncWrite(HaloDBOptions options) throws HaloDBException {\n String directory = TestUtils.getTestDirectory(\"HaloDBTest\", \"testSyncWrite\");\n\n options.setCompactionDisabled(true);\n options.setMaxFileSize(10 * 1024);\n options.enableSyncWrites(true);\n\n HaloDB db = getTestDB(directory, options);\n\n int noOfRecords = 10_000;\n List records = TestUtils.insertRandomRecords(db, noOfRecords);\n List updated = TestUtils.updateRecords(db, records);\n\n List actual = new ArrayList<>();\n db.newIterator().forEachRemaining(actual::add);\n\n Assert.assertTrue(actual.containsAll(updated) && updated.containsAll(actual));\n\n updated.forEach(record -> {\n try {\n byte[] value = db.get(record.getKey());\n Assert.assertEquals(record.getValue(), value);\n } catch (HaloDBException e) {\n throw new RuntimeException(e);\n }\n });\n }\n\n @Test(dataProvider = \"Options\")\n public void testToCheckThatLatestUpdateIsPickedAfterDBOpen(HaloDBOptions options) throws HaloDBException {\n\n String directory = TestUtils.getTestDirectory(\"HaloDBTest\", \"testToCheckThatLatestUpdateIsPickedAfterDBOpen\");\n\n options.setCompactionDisabled(true);\n\n // sized to ensure that there will be two files.\n options.setMaxFileSize(1500);\n\n HaloDB db = getTestDB(directory, options);\n\n byte[] key = TestUtils.generateRandomByteArray(7);\n byte[] value = null;\n\n // update the same record 100 times.\n // each key-value pair with the metadata is 20 bytes.\n // therefore 20 * 100 = 2000 bytes\n for (int i = 0; i < 100; i++) {\n value = TestUtils.generateRandomByteArray(7);\n db.put(key, value);\n }\n\n db.close();\n\n // open and read contents again.\n HaloDB openAgainDB = getTestDBWithoutDeletingFiles(directory, options);\n\n List actual = new ArrayList<>();\n openAgainDB.newIterator().forEachRemaining(actual::add);\n\n Assert.assertTrue(actual.size() == 1);\n\n Assert.assertEquals(openAgainDB.get(key), value);\n }\n\n @Test(dataProvider = \"Options\")\n public void testToCheckDelete(HaloDBOptions options) throws HaloDBException {\n String directory = TestUtils.getTestDirectory(\"HaloDBTest\", \"testToCheckDelete\");\n\n options.setCompactionDisabled(true);\n options.setMaxFileSize(10 * 1024);\n\n HaloDB db = getTestDB(directory, options);\n\n int noOfRecords = 10_000;\n List records = TestUtils.insertRandomRecords(db, noOfRecords);\n\n List deleted = new ArrayList<>();\n for (int i = 0; i < noOfRecords; i++) {\n if (i % 10 == 0) deleted.add(records.get(i));\n }\n\n TestUtils.deleteRecords(db, deleted);\n\n List remaining = new ArrayList<>();\n db.newIterator().forEachRemaining(remaining::add);\n\n Assert.assertTrue(remaining.size() == noOfRecords - deleted.size());\n\n deleted.forEach(r -> {\n try {\n Assert.assertNull(db.get(r.getKey()));\n } catch (HaloDBException e) {\n throw new RuntimeException(e);\n }\n });\n }\n\n @Test(dataProvider = \"Options\")\n public void testDeleteCloseAndOpen(HaloDBOptions options) throws HaloDBException {\n String directory = TestUtils.getTestDirectory(\"HaloDBTest\", \"testDeleteCloseAndOpen\");\n\n options.setCompactionDisabled(true);\n options.setMaxFileSize(10 * 1024);\n\n HaloDB db = getTestDB(directory, options);\n\n int noOfRecords = 10_000;\n List records = TestUtils.insertRandomRecords(db, noOfRecords);\n\n List deleted = new ArrayList<>();\n for (int i = 0; i < noOfRecords; i++) {\n if (i % 10 == 0) deleted.add(records.get(i));\n }\n\n TestUtils.deleteRecords(db, deleted);\n\n db.close();\n\n HaloDB openAgainDB = getTestDBWithoutDeletingFiles(directory, options);\n\n List remaining = new ArrayList<>();\n openAgainDB.newIterator().forEachRemaining(remaining::add);\n\n Assert.assertTrue(remaining.size() == noOfRecords - deleted.size());\n\n deleted.forEach(r -> {\n try {\n Assert.assertNull(openAgainDB.get(r.getKey()));\n } catch (HaloDBException e) {\n throw new RuntimeException(e);\n }\n });\n }\n\n @Test(dataProvider = \"Options\")\n public void testDeleteAndInsert(HaloDBOptions options) throws HaloDBException {\n String directory = TestUtils.getTestDirectory(\"HaloDBTest\", \"testDeleteAndInsert\");\n\n options.setCompactionDisabled(true);\n options.setMaxFileSize(10 * 1024);\n\n HaloDB db = getTestDB(directory, options);\n\n int noOfRecords = 10_000;\n List records = TestUtils.insertRandomRecords(db, noOfRecords);\n\n List deleted = new ArrayList<>();\n for (int i = 0; i < noOfRecords; i++) {\n if (i % 10 == 0) deleted.add(records.get(i));\n }\n\n TestUtils.deleteRecords(db, deleted);\n\n List deleteAndInsert = new ArrayList<>();\n deleted.forEach(r -> {\n try {\n byte[] value = TestUtils.generateRandomByteArray();\n db.put(r.getKey(), value);\n deleteAndInsert.add(new Record(r.getKey(), value));\n } catch (HaloDBException e) {\n throw new RuntimeException(e);\n }\n });\n\n\n List remaining = new ArrayList<>();\n db.newIterator().forEachRemaining(remaining::add);\n\n Assert.assertTrue(remaining.size() == noOfRecords);\n\n deleteAndInsert.forEach(r -> {\n try {\n Assert.assertEquals(r.getValue(), db.get(r.getKey()));\n } catch (HaloDBException e) {\n throw new RuntimeException(e);\n }\n });\n }\n\n @Test(dataProvider = \"Options\")\n public void testDeleteInsertCloseAndOpen(HaloDBOptions options) throws HaloDBException {\n String directory = TestUtils.getTestDirectory(\"tmp\", \"testDeleteInsertCloseAndOpen\");\n\n options.setCompactionDisabled(true);\n options.setMaxFileSize(10 * 1024);\n\n HaloDB db = getTestDB(directory, options);\n\n int noOfRecords = 10_000;\n List records = TestUtils.insertRandomRecords(db, noOfRecords);\n\n List deleted = new ArrayList<>();\n for (int i = 0; i < noOfRecords; i++) {\n if (i % 10 == 0) deleted.add(records.get(i));\n }\n\n TestUtils.deleteRecords(db, deleted);\n\n // inserting deleted records again. \n List deleteAndInsert = new ArrayList<>();\n deleted.forEach(r -> {\n try {\n byte[] value = TestUtils.generateRandomByteArray();\n db.put(r.getKey(), value);\n deleteAndInsert.add(new Record(r.getKey(), value));\n } catch (HaloDBException e) {\n throw new RuntimeException(e);\n }\n });\n\n db.close();\n HaloDB openAgainDB = getTestDBWithoutDeletingFiles(directory, options);\n\n List remaining = new ArrayList<>();\n openAgainDB.newIterator().forEachRemaining(remaining::add);\n\n Assert.assertTrue(remaining.size() == noOfRecords);\n\n // make sure that records that were earlier deleted shows up now, since they were put back later.\n deleteAndInsert.forEach(r -> {\n try {\n Assert.assertEquals(r.getValue(), openAgainDB.get(r.getKey()));\n } catch (HaloDBException e) {\n throw new RuntimeException(e);\n }\n });\n }\n\n @Test\n public void testDBMetaFile() throws HaloDBException, IOException {\n String directory = TestUtils.getTestDirectory(\"HaloDBTest\", \"testDBMetaFile\");\n\n HaloDBOptions options = new HaloDBOptions();\n int maxFileSize = 1024 * 1024 * 1024;\n options.setMaxFileSize(maxFileSize);\n HaloDB db = getTestDB(directory, options);\n\n // Make sure that the META file was written.\n Assert.assertTrue(Paths.get(directory, DBMetaData.METADATA_FILE_NAME).toFile().exists());\n\n DBMetaData metaData = new DBMetaData(dbDirectory);\n metaData.loadFromFileIfExists();\n\n // Make sure that the open flag was set on db open.\n Assert.assertTrue(metaData.isOpen());\n\n // Default value of ioError flag must be false. \n Assert.assertFalse(metaData.isIOError());\n\n // since we just created the db max file size should be set to one we set in HaloDBOptions\n Assert.assertEquals(metaData.getMaxFileSize(), maxFileSize);\n Assert.assertEquals(metaData.getVersion(), Versions.CURRENT_META_FILE_VERSION);\n\n db.close();\n\n // Make sure that the META file was written.\n Assert.assertTrue(Paths.get(directory, DBMetaData.METADATA_FILE_NAME).toFile().exists());\n\n // Make sure that the flags were set correctly on close.\n metaData.loadFromFileIfExists();\n\n Assert.assertEquals(metaData.getVersion(), Versions.CURRENT_META_FILE_VERSION);\n Assert.assertFalse(metaData.isOpen());\n Assert.assertFalse(metaData.isIOError());\n Assert.assertEquals(metaData.getMaxFileSize(), maxFileSize);\n }\n\n @Test(expectedExceptions = IllegalArgumentException.class, expectedExceptionsMessageRegExp = \"File size cannot be changed after db was created.*\")\n public void testMaxFileSize() throws HaloDBException, IOException {\n String directory = TestUtils.getTestDirectory(\"HaloDBTest\", \"testMaxFileSize\");\n\n HaloDBOptions options = new HaloDBOptions();\n int maxFileSize = 1024 * 1024 * 1024;\n options.setMaxFileSize(maxFileSize);\n HaloDB db = getTestDB(directory, options);\n\n DBMetaData metaData = new DBMetaData(dbDirectory);\n metaData.loadFromFileIfExists();\n\n Assert.assertEquals(metaData.getMaxFileSize(), maxFileSize);\n\n db.close();\n\n // try opening the db with max file size changed.\n options.setMaxFileSize(500 * 1024 * 1024);\n getTestDBWithoutDeletingFiles(directory, options);\n }\n\n @Test(expectedExceptions = HaloDBException.class, expectedExceptionsMessageRegExp = \"Another process already holds a lock for this db.\")\n public void testLock() throws Throwable {\n String directory = TestUtils.getTestDirectory(\"HaloDBTest\", \"testLock\");\n\n HaloDB db = getTestDB(directory, new HaloDBOptions());\n db.resetStats();\n HaloDB anotherDB = HaloDB.open(directory, new HaloDBOptions());\n anotherDB.resetStats();\n }\n\n @Test\n public void testLockReleaseOnError() throws Throwable {\n\n new MockUp() {\n int count = 0;\n\n @Mock\n void loadFromFileIfExists() throws IOException {\n System.out.println(\"Mock called\");\n if (count == 0) {\n // throw an exception the first time the method is called.\n count = 1;\n throw new IOException();\n }\n }\n };\n\n String directory = TestUtils.getTestDirectory(\"HaloDBTest\", \"testLockReleaseOnError\");\n\n HaloDB db = null;\n try {\n db = getTestDB(directory, new HaloDBOptions());\n } catch (HaloDBException e) {\n // swallow the mocked exception. \n }\n // make sure open() failed.\n Assert.assertNull(db);\n\n // the lock should have been released when previous open() failed. \n HaloDB anotherDB = getTestDBWithoutDeletingFiles(directory, new HaloDBOptions());\n anotherDB.put(Longs.toByteArray(1), Longs.toByteArray(1));\n Assert.assertEquals(anotherDB.size(), 1);\n }\n\n @Test(expectedExceptions = HaloDBException.class)\n public void testPutAfterClose() throws HaloDBException {\n String directory = TestUtils.getTestDirectory(\"HaloDBTest\", \"testPutAfterClose\");\n HaloDB db = getTestDB(directory, new HaloDBOptions());\n db.put(Longs.toByteArray(1), Longs.toByteArray(1));\n db.close();\n db.put(Longs.toByteArray(2), Longs.toByteArray(2));\n }\n\n @Test(expectedExceptions = HaloDBException.class)\n public void testDeleteAfterClose() throws HaloDBException {\n String directory = TestUtils.getTestDirectory(\"HaloDBTest\", \"testDeleteAfterClose\");\n\n HaloDB db = getTestDB(directory, new HaloDBOptions());\n db.put(Longs.toByteArray(1), Longs.toByteArray(1));\n db.put(Longs.toByteArray(2), Longs.toByteArray(2));\n db.delete(Longs.toByteArray(1));\n db.close();\n db.delete(Longs.toByteArray(2));\n }\n\n @Test(expectedExceptions = NullPointerException.class)\n public void testPutAfterCloseWithoutWrites() throws HaloDBException {\n String directory = TestUtils.getTestDirectory(\"HaloDBTest\", \"testPutAfterCloseWithoutWrites\");\n HaloDB db = getTestDB(directory, new HaloDBOptions());\n db.close();\n db.put(Longs.toByteArray(1), Longs.toByteArray(1));\n }\n\n @Test(expectedExceptions = NullPointerException.class)\n public void testDeleteAfterCloseWithoutWrites() throws HaloDBException {\n String directory = TestUtils.getTestDirectory(\"HaloDBTest\", \"testDeleteAfterCloseWithoutWrites\");\n\n HaloDB db = getTestDB(directory, new HaloDBOptions());\n db.put(Longs.toByteArray(1), Longs.toByteArray(1));\n Assert.assertEquals(db.get(Longs.toByteArray(1)), Longs.toByteArray(1));\n db.close();\n db.delete(Longs.toByteArray(1));\n }\n\n @Test\n public void testSnapshot() throws HaloDBException {\n String directory = TestUtils.getTestDirectory(\"HaloDBTest\", \"testSnapshot\");\n HaloDBOptions options = new HaloDBOptions();\n // Generate several data files\n options.setMaxFileSize(10000);\n HaloDB db = getTestDB(directory, options);\n for (int i = 10000; i < 10000 + 10*1000; i++) {\n db.put(ByteBuffer.allocate(4).putInt(i).array(), ByteBuffer.allocate(8).putInt(i).putInt(i*1024).array());\n }\n db.snapshot();\n\n String snapshotDir = db.getSnapshotDirectory().toString();\n HaloDB snapshotDB = getTestDBWithoutDeletingFiles(snapshotDir, options);\n\n for (int i = 10000; i < 10000 + 10*1000; i++) {\n byte[] value = snapshotDB.get(ByteBuffer.allocate(4).putInt(i).array());\n Assert.assertTrue(Arrays.equals(value, ByteBuffer.allocate(8).putInt(i).putInt(i*1024).array()));\n }\n\n snapshotDB.close();\n db.close();\n }\n\n @Test\n public void testCreateAndDeleteSnapshot() throws HaloDBException {\n String directory = TestUtils.getTestDirectory(\"HaloDBTest\", \"testCreateAndDeleteSnapshot\");\n HaloDBOptions options = new HaloDBOptions();\n // Generate several data files\n options.setMaxFileSize(10000);\n HaloDB db = getTestDB(directory, options);\n for (int i = 10000; i < 10000 + 10*1000; i++) {\n db.put(ByteBuffer.allocate(4).putInt(i).array(), ByteBuffer.allocate(8).putInt(i).putInt(i*1024).array());\n }\n\n Assert.assertTrue(db.clearSnapshot());\n db.snapshot();\n\n Assert.assertTrue(db.clearSnapshot());\n\n File snapshotDir = db.getSnapshotDirectory();\n Assert.assertFalse(snapshotDir.exists());\n\n db.close();\n }\n\n @Test\n public void testSnapshotAfterBeenCompacted() throws HaloDBException {\n String directory = TestUtils.getTestDirectory(\"HaloDBTest\", \"testSnapshotAfterBeenCompacted\");\n HaloDBOptions options = new HaloDBOptions();\n // Generate several data files\n options.setMaxFileSize(10000);\n options.setCompactionThresholdPerFile(0.7);\n HaloDB db = getTestDB(directory, options);\n for (int i = 10000; i < 10000 + 10*1000; i++) {\n db.put(ByteBuffer.allocate(4).putInt(i).array(), ByteBuffer.allocate(8).putInt(i).putInt(i*1024).array());\n }\n db.snapshot();\n\n // overwrite all previous record\n for (int i = 10000; i < 10000 + 10*1000; i++) {\n db.put(ByteBuffer.allocate(4).putInt(i).array(), ByteBuffer.allocate(8).putInt(i).putInt(i*2048).array());\n }\n TestUtils.waitForCompactionToComplete(db);\n\n String snapshotDir = db.getSnapshotDirectory().toString();\n HaloDB snapshotDB = getTestDBWithoutDeletingFiles(snapshotDir, options);\n\n for (int i = 10000; i < 10000 + 10*1000; i++) {\n byte[] key = ByteBuffer.allocate(4).putInt(i).array();\n byte[] value = snapshotDB.get(key);\n Assert.assertTrue(Arrays.equals(value, ByteBuffer.allocate(8).putInt(i).putInt(i*1024).array()));\n Assert.assertFalse(Arrays.equals(value, db.get(key)));\n }\n\n snapshotDB.close();\n db.close();\n }\n\n}\n" }, { "path": "src/test/java/com/oath/halodb/HashTableTestUtils.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\n// This code is a derivative work heavily modified from the OHC project. See NOTICE file for copyright and license.\n\npackage com.oath.halodb;\n\nimport com.google.common.base.Charsets;\nimport com.google.common.primitives.Longs;\n\nimport org.testng.Assert;\n\nimport java.nio.ByteBuffer;\nimport java.util.ArrayList;\nimport java.util.List;\nimport java.util.Random;\n\nfinal class HashTableTestUtils\n{\n public static final long ONE_MB = 1024 * 1024;\n public static final HashTableValueSerializer stringSerializer = new HashTableValueSerializer()\n {\n public void serialize(String s, ByteBuffer buf)\n {\n byte[] bytes = s.getBytes(Charsets.UTF_8);\n buf.put((byte) ((bytes.length >>> 8) & 0xFF));\n buf.put((byte) ((bytes.length >>> 0) & 0xFF));\n buf.put(bytes);\n }\n\n public String deserialize(ByteBuffer buf)\n {\n int length = (((buf.get() & 0xff) << 8) + ((buf.get() & 0xff) << 0));\n byte[] bytes = new byte[length];\n buf.get(bytes);\n return new String(bytes, Charsets.UTF_8);\n }\n\n public int serializedSize(String s)\n {\n return writeUTFLen(s);\n }\n };\n\n public static final HashTableValueSerializer byteArraySerializer = new HashTableValueSerializer()\n {\n @Override\n public void serialize(byte[] value, ByteBuffer buf) {\n buf.put(value);\n }\n\n @Override\n public byte[] deserialize(ByteBuffer buf) {\n // Cannot use buf.array() as buf is read-only for get() operations.\n byte[] array = new byte[buf.remaining()];\n buf.get(array);\n return array;\n }\n\n @Override\n public int serializedSize(byte[] value) {\n return value.length;\n }\n };\n\n public static final HashTableValueSerializer stringSerializerFailSerialize = new HashTableValueSerializer()\n {\n public void serialize(String s, ByteBuffer buf)\n {\n throw new RuntimeException(\"foo bar\");\n }\n\n public String deserialize(ByteBuffer buf)\n {\n int length = (buf.get() << 8) + (buf.get() << 0);\n byte[] bytes = new byte[length];\n buf.get(bytes);\n return new String(bytes, Charsets.UTF_8);\n }\n\n public int serializedSize(String s)\n {\n return writeUTFLen(s);\n }\n };\n public static final HashTableValueSerializer stringSerializerFailDeserialize = new HashTableValueSerializer()\n {\n public void serialize(String s, ByteBuffer buf)\n {\n byte[] bytes = s.getBytes(Charsets.UTF_8);\n buf.put((byte) ((bytes.length >>> 8) & 0xFF));\n buf.put((byte) ((bytes.length >>> 0) & 0xFF));\n buf.put(bytes);\n }\n\n public String deserialize(ByteBuffer buf)\n {\n throw new RuntimeException(\"foo bar\");\n }\n\n public int serializedSize(String s)\n {\n return writeUTFLen(s);\n }\n };\n\n public static final HashTableValueSerializer byteArraySerializerFailSerialize = new HashTableValueSerializer()\n {\n public void serialize(byte[] s, ByteBuffer buf)\n {\n throw new RuntimeException(\"foo bar\");\n }\n\n public byte[] deserialize(ByteBuffer buf)\n {\n byte[] array = new byte[buf.remaining()];\n buf.get(array);\n return array;\n }\n\n public int serializedSize(byte[] s)\n {\n return s.length;\n }\n };\n\n static int writeUTFLen(String str)\n {\n int strlen = str.length();\n int utflen = 0;\n int c;\n\n for (int i = 0; i < strlen; i++)\n {\n c = str.charAt(i);\n if ((c >= 0x0001) && (c <= 0x007F))\n utflen++;\n else if (c > 0x07FF)\n utflen += 3;\n else\n utflen += 2;\n }\n\n if (utflen > 65535)\n throw new RuntimeException(\"encoded string too long: \" + utflen + \" bytes\");\n\n return utflen + 2;\n }\n\n public static final byte[] dummyByteArray;\n public static final HashTableValueSerializer intSerializer = new HashTableValueSerializer()\n {\n public void serialize(Integer s, ByteBuffer buf)\n {\n buf.put((byte)(1 & 0xff));\n buf.putChar('A');\n buf.putDouble(42.42424242d);\n buf.putFloat(11.111f);\n buf.putInt(s);\n buf.putLong(Long.MAX_VALUE);\n buf.putShort((short)(0x7654 & 0xFFFF));\n buf.put(dummyByteArray);\n }\n\n public Integer deserialize(ByteBuffer buf)\n {\n Assert.assertEquals(buf.get(), (byte) 1);\n Assert.assertEquals(buf.getChar(), 'A');\n Assert.assertEquals(buf.getDouble(), 42.42424242d);\n Assert.assertEquals(buf.getFloat(), 11.111f);\n int r = buf.getInt();\n Assert.assertEquals(buf.getLong(), Long.MAX_VALUE);\n Assert.assertEquals(buf.getShort(), 0x7654);\n byte[] b = new byte[dummyByteArray.length];\n buf.get(b);\n Assert.assertEquals(b, dummyByteArray);\n return r;\n }\n\n public int serializedSize(Integer s)\n {\n return 529;\n }\n };\n public static final HashTableValueSerializer intSerializerFailSerialize = new HashTableValueSerializer()\n {\n public void serialize(Integer s, ByteBuffer buf)\n {\n throw new RuntimeException(\"foo bar\");\n }\n\n public Integer deserialize(ByteBuffer buf)\n {\n Assert.assertEquals(buf.get(), (byte) 1);\n Assert.assertEquals(buf.getChar(), 'A');\n Assert.assertEquals(buf.getDouble(), 42.42424242d);\n Assert.assertEquals(buf.getFloat(), 11.111f);\n int r = buf.getInt();\n Assert.assertEquals(buf.getLong(), Long.MAX_VALUE);\n Assert.assertEquals(buf.getShort(), 0x7654);\n byte[] b = new byte[dummyByteArray.length];\n buf.get(b);\n Assert.assertEquals(b, dummyByteArray);\n return r;\n }\n\n public int serializedSize(Integer s)\n {\n return 529;\n }\n };\n public static final HashTableValueSerializer intSerializerFailDeserialize = new HashTableValueSerializer()\n {\n public void serialize(Integer s, ByteBuffer buf)\n {\n buf.putInt(s);\n }\n\n public Integer deserialize(ByteBuffer buf)\n {\n throw new RuntimeException(\"foo bar\");\n }\n\n public int serializedSize(Integer s)\n {\n return 4;\n }\n };\n static final String big;\n static final String bigRandom;\n\n static {\n dummyByteArray = new byte[500];\n for (int i = 0; i < HashTableTestUtils.dummyByteArray.length; i++)\n HashTableTestUtils.dummyByteArray[i] = (byte) ((byte) i % 199);\n }\n\n static int manyCount = 20000;\n\n static\n {\n\n StringBuilder sb = new StringBuilder();\n for (int i = 0; i < 1000; i++)\n sb.append(\"the quick brown fox jumps over the lazy dog\");\n big = sb.toString();\n\n Random r = new Random();\n sb.setLength(0);\n for (int i = 0; i < 30000; i++)\n sb.append((char) (r.nextInt(99) + 31));\n bigRandom = sb.toString();\n }\n\n static List fillMany(OffHeapHashTable cache, int fixedValueSize)\n {\n return fill(cache, fixedValueSize, manyCount);\n }\n\n static List fill(OffHeapHashTable cache, int fixedValueSize, int count)\n {\n List many = new ArrayList<>();\n for (int i = 0; i < count; i++) {\n byte[] key = Longs.toByteArray(i);\n byte[] value = HashTableTestUtils.randomBytes(fixedValueSize);\n cache.put(key, value);\n many.add(new KeyValuePair(key, value));\n }\n\n return many;\n }\n\n static byte[] randomBytes(int len)\n {\n Random r = new Random();\n byte[] arr = new byte[len];\n r.nextBytes(arr);\n return arr;\n }\n\n static class KeyValuePair {\n byte[] key, value;\n\n KeyValuePair(byte[] key, byte[] value) {\n this.key = key;\n this.value = value;\n }\n }\n}\n" }, { "path": "src/test/java/com/oath/halodb/HashTableUtilTest.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\n// This code is a derivative work heavily modified from the OHC project. See NOTICE file for copyright and license.\n\npackage com.oath.halodb;\n\nimport com.oath.halodb.HashTableUtil;\n\nimport org.testng.Assert;\nimport org.testng.annotations.Test;\n\npublic class HashTableUtilTest\n{\n static final long BIG = 2L << 40;\n\n @Test\n public void testBitNum()\n {\n Assert.assertEquals(HashTableUtil.bitNum(0), 0);\n Assert.assertEquals(HashTableUtil.bitNum(1), 1);\n Assert.assertEquals(HashTableUtil.bitNum(2), 2);\n Assert.assertEquals(HashTableUtil.bitNum(4), 3);\n Assert.assertEquals(HashTableUtil.bitNum(8), 4);\n Assert.assertEquals(HashTableUtil.bitNum(16), 5);\n Assert.assertEquals(HashTableUtil.bitNum(32), 6);\n Assert.assertEquals(HashTableUtil.bitNum(64), 7);\n Assert.assertEquals(HashTableUtil.bitNum(128), 8);\n Assert.assertEquals(HashTableUtil.bitNum(256), 9);\n Assert.assertEquals(HashTableUtil.bitNum(1024), 11);\n Assert.assertEquals(HashTableUtil.bitNum(65536), 17);\n }\n}\n" }, { "path": "src/test/java/com/oath/halodb/HashTableValueSerializerTest.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\n// This code is a derivative work heavily modified from the OHC project. See NOTICE file for copyright and license.\n\npackage com.oath.halodb;\n\nimport com.google.common.primitives.Ints;\nimport com.google.common.primitives.Longs;\n\nimport org.testng.Assert;\nimport org.testng.annotations.AfterMethod;\nimport org.testng.annotations.Test;\n\nimport java.io.IOException;\n\npublic class HashTableValueSerializerTest\n{\n @AfterMethod(alwaysRun = true)\n public void deinit()\n {\n Uns.clearUnsDebugForTest();\n }\n\n @Test\n public void testFailingValueSerializerOnPut() throws IOException, InterruptedException\n {\n try (OffHeapHashTable cache = OffHeapHashTableBuilder.newBuilder()\n .valueSerializer(HashTableTestUtils.byteArraySerializerFailSerialize)\n .fixedValueSize(8)\n .build())\n {\n try\n {\n cache.put(Ints.toByteArray(1), Longs.toByteArray(1));\n Assert.fail();\n }\n catch (RuntimeException ignored)\n {\n // ok\n }\n\n try\n {\n cache.putIfAbsent(Ints.toByteArray(1), Longs.toByteArray(1));\n Assert.fail();\n }\n catch (RuntimeException ignored)\n {\n // ok\n }\n\n try\n {\n cache.addOrReplace(Ints.toByteArray(1), Longs.toByteArray(1), Longs.toByteArray(2));\n Assert.fail();\n }\n catch (RuntimeException ignored)\n {\n // ok\n }\n\n }\n }\n}\n" }, { "path": "src/test/java/com/oath/halodb/HasherTest.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\n// This code is a derivative work heavily modified from the OHC project. See NOTICE file for copyright and license.\n\npackage com.oath.halodb;\n\nimport org.testng.Assert;\nimport org.testng.annotations.Test;\n\nimport java.util.Random;\n\npublic class HasherTest\n{\n @Test\n public void testMurmur3()\n {\n test(HashAlgorithm.MURMUR3);\n }\n\n @Test\n public void testCRC32()\n {\n test(HashAlgorithm.CRC32);\n }\n\n @Test\n public void testXX()\n {\n test(HashAlgorithm.XX);\n }\n\n private void test(HashAlgorithm hash)\n {\n Random rand = new Random();\n\n byte[] buf = new byte[3211];\n rand.nextBytes(buf);\n\n Hasher hasher = Hasher.create(hash);\n long arrayVal = hasher.hash(buf);\n long memAddr = Uns.allocate(buf.length + 99);\n try\n {\n Uns.copyMemory(buf, 0, memAddr, 99L, buf.length);\n\n long memoryVal = hasher.hash(memAddr, 99L, buf.length);\n\n Assert.assertEquals(memoryVal, arrayVal);\n }\n finally\n {\n Uns.free(memAddr);\n }\n }\n}\n" }, { "path": "src/test/java/com/oath/halodb/IndexFileEntryTest.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport org.testng.Assert;\nimport org.testng.annotations.Test;\n\nimport java.nio.ByteBuffer;\nimport java.util.zip.CRC32;\n\nimport static com.oath.halodb.IndexFileEntry.*;\n\npublic class IndexFileEntryTest {\n\n @Test\n public void serializeIndexFileEntry() {\n byte[] key = TestUtils.generateRandomByteArray(8);\n int recordSize = 1024;\n int recordOffset = 10240;\n byte keySize = (byte) key.length;\n long sequenceNumber = 100;\n int version = 200;\n\n ByteBuffer header = ByteBuffer.allocate(INDEX_FILE_HEADER_SIZE);\n header.put(VERSION_OFFSET, (byte)version);\n header.put(KEY_SIZE_OFFSET, keySize);\n header.putInt(RECORD_SIZE_OFFSET, recordSize);\n header.putInt(RECORD_OFFSET, recordOffset);\n header.putLong(SEQUENCE_NUMBER_OFFSET, sequenceNumber);\n\n CRC32 crc32 = new CRC32();\n crc32.update(header.array(), VERSION_OFFSET, INDEX_FILE_HEADER_SIZE-CHECKSUM_SIZE);\n crc32.update(key);\n long checkSum = crc32.getValue();\n header.putInt(CHECKSUM_OFFSET, Utils.toSignedIntFromLong(checkSum));\n\n IndexFileEntry entry = new IndexFileEntry(key, recordSize, recordOffset, sequenceNumber, version, -1);\n ByteBuffer[] buffers = entry.serialize();\n\n Assert.assertEquals(header, buffers[0]);\n Assert.assertEquals(ByteBuffer.wrap(key), buffers[1]);\n }\n\n @Test\n public void deserializeIndexFileEntry() {\n byte[] key = TestUtils.generateRandomByteArray(8);\n int recordSize = 1024;\n int recordOffset = 10240;\n byte keySize = (byte) key.length;\n long sequenceNumber = 100;\n int version = 10;\n long checksum = 42323;\n\n ByteBuffer header = ByteBuffer.allocate(IndexFileEntry.INDEX_FILE_HEADER_SIZE + keySize);\n header.putInt((int)checksum);\n header.put((byte)version);\n header.put(keySize);\n header.putInt(recordSize);\n header.putInt(recordOffset);\n header.putLong(sequenceNumber);\n header.put(key);\n header.flip();\n\n IndexFileEntry entry = IndexFileEntry.deserialize(header);\n\n Assert.assertEquals(entry.getCheckSum(), checksum);\n Assert.assertEquals(entry.getVersion(), version);\n Assert.assertEquals(entry.getRecordSize(), recordSize);\n Assert.assertEquals(entry.getRecordOffset(), recordOffset);\n Assert.assertEquals(entry.getSequenceNumber(), sequenceNumber);\n Assert.assertEquals(entry.getKey(), key);\n }\n}\n" }, { "path": "src/test/java/com/oath/halodb/KeyBufferTest.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\n// This code is a derivative work heavily modified from the OHC project. See NOTICE file for copyright and license.\n\npackage com.oath.halodb;\n\nimport com.google.common.hash.HashCode;\nimport com.google.common.hash.Hasher;\nimport com.google.common.hash.Hashing;\n\nimport org.testng.annotations.AfterMethod;\nimport org.testng.annotations.DataProvider;\nimport org.testng.annotations.Test;\n\nimport java.nio.ByteBuffer;\n\nimport static org.testng.Assert.assertEquals;\nimport static org.testng.Assert.assertFalse;\nimport static org.testng.Assert.assertTrue;\n\npublic class KeyBufferTest\n{\n @AfterMethod(alwaysRun = true)\n public void deinit()\n {\n Uns.clearUnsDebugForTest();\n }\n\n @DataProvider\n public Object[][] hashes()\n {\n return new Object[][]{\n { HashAlgorithm.MURMUR3 },\n { HashAlgorithm.CRC32 },\n // TODO { HashAlgorithm.XX }\n };\n }\n\n @Test(dataProvider = \"hashes\")\n public void testHashFinish(HashAlgorithm hashAlgorithm) throws Exception\n {\n\n ByteBuffer buf = ByteBuffer.allocate(12);\n byte[] ref = TestUtils.generateRandomByteArray(10);\n buf.put((byte) (42 & 0xff));\n buf.put(ref);\n buf.put((byte) (0xf0 & 0xff));\n\n KeyBuffer out = new KeyBuffer(buf.array());\n out.finish(com.oath.halodb.Hasher.create(hashAlgorithm));\n\n Hasher hasher = hasher(hashAlgorithm);\n hasher.putByte((byte) 42);\n hasher.putBytes(ref);\n hasher.putByte((byte) 0xf0);\n long longHash = hash(hasher);\n\n assertEquals(out.hash(), longHash);\n }\n\n private long hash(Hasher hasher)\n {\n HashCode hash = hasher.hash();\n if (hash.bits() == 32)\n {\n long longHash = hash.asInt();\n longHash = longHash << 32 | (longHash & 0xffffffffL);\n return longHash;\n }\n return hash.asLong();\n }\n\n private Hasher hasher(HashAlgorithm hashAlgorithm)\n {\n switch (hashAlgorithm)\n {\n case MURMUR3:\n return Hashing.murmur3_128().newHasher();\n case CRC32:\n return Hashing.crc32().newHasher();\n default:\n throw new IllegalArgumentException();\n }\n }\n\n @Test(dataProvider = \"hashes\", dependsOnMethods = \"testHashFinish\")\n public void testHashFinish16(HashAlgorithm hashAlgorithm) throws Exception\n {\n\n byte[] ref = TestUtils.generateRandomByteArray(14);\n ByteBuffer buf = ByteBuffer.allocate(16);\n buf.put((byte) (42 & 0xff));\n buf.put(ref);\n buf.put((byte) (0xf0 & 0xff));\n KeyBuffer out = new KeyBuffer(buf.array());\n out.finish(com.oath.halodb.Hasher.create(hashAlgorithm));\n\n Hasher hasher = hasher(hashAlgorithm);\n hasher.putByte((byte) 42);\n hasher.putBytes(ref);\n hasher.putByte((byte) 0xf0);\n long longHash = hash(hasher);\n\n assertEquals(out.hash(), longHash);\n }\n\n @Test(dataProvider = \"hashes\", dependsOnMethods = \"testHashFinish16\")\n public void testHashRandom(HashAlgorithm hashAlgorithm) throws Exception\n {\n for (int i = 1; i < 4100; i++)\n {\n for (int j = 0; j < 10; j++)\n {\n\n byte[] ref = TestUtils.generateRandomByteArray(i);\n ByteBuffer buf = ByteBuffer.allocate(i);\n buf.put(ref);\n KeyBuffer out = new KeyBuffer(buf.array());\n out.finish(com.oath.halodb.Hasher.create(hashAlgorithm));\n\n Hasher hasher = hasher(hashAlgorithm);\n hasher.putBytes(ref);\n long longHash = hash(hasher);\n\n assertEquals(out.hash(), longHash);\n }\n }\n }\n\n @Test\n public void testSameKey() {\n\n int keyLength = 8;\n byte[] randomKey = TestUtils.generateRandomByteArray(keyLength);\n compareKey(randomKey);\n\n keyLength = 9;\n randomKey = TestUtils.generateRandomByteArray(keyLength);\n compareKey(randomKey);\n\n for (int i = 0; i < 128; i++) {\n randomKey = TestUtils.generateRandomByteArray();\n keyLength = randomKey.length;\n if (keyLength == 0)\n continue;\n compareKey(randomKey);\n }\n\n }\n\n private void compareKey(byte[] randomKey) {\n\n long adr = Uns.allocate(NonMemoryPoolHashEntries.ENTRY_OFF_DATA + randomKey.length, true);\n try {\n KeyBuffer key = new KeyBuffer(randomKey);\n key.finish(com.oath.halodb.Hasher.create(HashAlgorithm.MURMUR3));\n\n NonMemoryPoolHashEntries.init(randomKey.length, adr);\n Uns.setMemory(adr, NonMemoryPoolHashEntries.ENTRY_OFF_DATA, randomKey.length, (byte) 0);\n\n assertFalse(key.sameKey(adr));\n\n Uns.copyMemory(randomKey, 0, adr, NonMemoryPoolHashEntries.ENTRY_OFF_DATA, randomKey.length);\n NonMemoryPoolHashEntries.init(randomKey.length, adr);\n assertTrue(key.sameKey(adr));\n } finally {\n Uns.free(adr);\n }\n }\n}\n" }, { "path": "src/test/java/com/oath/halodb/LinkedImplTest.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\n// This code is a derivative work heavily modified from the OHC project. See NOTICE file for copyright and license.\n\npackage com.oath.halodb;\n\nimport org.testng.annotations.AfterMethod;\nimport org.testng.annotations.Test;\n\nimport java.io.IOException;\n\npublic class LinkedImplTest\n{\n @AfterMethod(alwaysRun = true)\n public void deinit()\n {\n Uns.clearUnsDebugForTest();\n }\n\n static OffHeapHashTable cache()\n {\n return cache(256);\n }\n\n static OffHeapHashTable cache(long capacity)\n {\n return cache(capacity, -1);\n }\n\n static OffHeapHashTable cache(long capacity, int hashTableSize)\n {\n return cache(capacity, hashTableSize, -1, -1);\n }\n\n static OffHeapHashTable cache(long capacity, int hashTableSize, int segments, long maxEntrySize)\n {\n OffHeapHashTableBuilder builder = OffHeapHashTableBuilder.newBuilder()\n .valueSerializer(HashTableTestUtils.stringSerializer);\n if (hashTableSize > 0)\n builder.hashTableSize(hashTableSize);\n if (segments > 0)\n builder.segmentCount(segments);\n else\n // use 16 segments by default to prevent differing test behaviour on varying test hardware\n builder.segmentCount(16);\n\n return builder.build();\n }\n\n @Test(expectedExceptions = IllegalArgumentException.class)\n public void testExtremeHashTableSize() throws IOException\n {\n OffHeapHashTableBuilder builder = OffHeapHashTableBuilder.newBuilder()\n .hashTableSize(1 << 30);\n builder.build().close();\n }\n\n}\n" }, { "path": "src/test/java/com/oath/halodb/LongArrayListTest.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\n// This code is a derivative work heavily modified from the OHC project. See NOTICE file for copyright and license.\n\npackage com.oath.halodb;\n\nimport com.oath.halodb.LongArrayList;\n\nimport org.testng.annotations.Test;\n\nimport static org.testng.Assert.assertEquals;\n\npublic class LongArrayListTest\n{\n @Test\n public void testLongArrayList()\n {\n LongArrayList l = new LongArrayList();\n\n assertEquals(l.size(), 0);\n\n l.add(0);\n assertEquals(l.size(), 1);\n\n for (int i=1;i<=20;i++)\n {\n l.add(i);\n assertEquals(l.size(), i + 1);\n }\n\n for (int i=0;i<=20;i++)\n {\n assertEquals(l.getLong(i), i);\n }\n }\n}" }, { "path": "src/test/java/com/oath/halodb/MemoryPoolChunkTest.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport com.google.common.primitives.Longs;\n\nimport org.testng.Assert;\nimport org.testng.annotations.AfterMethod;\nimport org.testng.annotations.Test;\n\nimport java.util.Random;\n\npublic class MemoryPoolChunkTest {\n\n private MemoryPoolChunk chunk = null;\n\n @AfterMethod(alwaysRun = true)\n private void destroyChunk() {\n if (chunk != null) {\n chunk.destroy();\n }\n }\n\n @Test\n public void testSetAndGetMethods() {\n int chunkSize = 16 * 1024;\n int fixedKeyLength = 12, fixedValueLength = 20;\n int slotSize = MemoryPoolHashEntries.HEADER_SIZE + fixedKeyLength + fixedValueLength;\n\n chunk = MemoryPoolChunk.create(chunkSize, fixedKeyLength, fixedValueLength);\n int offset = chunk.getWriteOffset();\n\n Assert.assertEquals(chunk.remaining(), chunkSize);\n Assert.assertEquals(chunk.getWriteOffset(), 0);\n\n // write to an empty slot.\n byte[] key = Longs.toByteArray(101);\n byte[] value = HashTableTestUtils.randomBytes(fixedValueLength);\n MemoryPoolAddress nextAddress = new MemoryPoolAddress((byte) 10, 34343);\n chunk.fillNextSlot(key, value, nextAddress);\n\n Assert.assertEquals(chunk.getWriteOffset(), offset + slotSize);\n Assert.assertEquals(chunk.remaining(), chunkSize-slotSize);\n Assert.assertTrue(chunk.compareKey(offset, key));\n Assert.assertTrue(chunk.compareValue(offset, value));\n\n MemoryPoolAddress actual = chunk.getNextAddress(offset);\n Assert.assertEquals(actual.chunkIndex, nextAddress.chunkIndex);\n Assert.assertEquals(actual.chunkOffset, nextAddress.chunkOffset);\n\n // write to the next empty slot.\n byte[] key2 = HashTableTestUtils.randomBytes(fixedKeyLength);\n byte[] value2 = HashTableTestUtils.randomBytes(fixedValueLength);\n MemoryPoolAddress nextAddress2 = new MemoryPoolAddress((byte) 0, 4454545);\n chunk.fillNextSlot(key2, value2, nextAddress2);\n Assert.assertEquals(chunk.getWriteOffset(), offset + 2*slotSize);\n Assert.assertEquals(chunk.remaining(), chunkSize-2*slotSize);\n\n offset += slotSize;\n Assert.assertTrue(chunk.compareKey(offset, key2));\n Assert.assertTrue(chunk.compareValue(offset, value2));\n\n actual = chunk.getNextAddress(offset);\n Assert.assertEquals(actual.chunkIndex, nextAddress2.chunkIndex);\n Assert.assertEquals(actual.chunkOffset, nextAddress2.chunkOffset);\n\n // update an existing slot.\n byte[] key3 = Longs.toByteArray(0x64735981289L);\n byte[] value3 = HashTableTestUtils.randomBytes(fixedValueLength);\n MemoryPoolAddress nextAddress3 = new MemoryPoolAddress((byte)-1, -1);\n chunk.fillSlot(0, key3, value3, nextAddress3);\n\n offset = 0;\n Assert.assertTrue(chunk.compareKey(offset, key3));\n Assert.assertTrue(chunk.compareValue(offset, value3));\n\n // write offset should remain unchanged.\n Assert.assertEquals(chunk.getWriteOffset(), offset + 2*slotSize);\n Assert.assertEquals(chunk.remaining(), chunkSize-2*slotSize);\n }\n\n @Test(expectedExceptions = IllegalArgumentException.class, expectedExceptionsMessageRegExp = \"Invalid offset.*\")\n public void testWithInvalidOffset() {\n int chunkSize = 256;\n int fixedKeyLength = 100, fixedValueLength = 100;\n MemoryPoolAddress next = new MemoryPoolAddress((byte)-1, -1);\n chunk = MemoryPoolChunk.create(chunkSize, fixedKeyLength, fixedValueLength);\n chunk.fillSlot(chunkSize - 5, HashTableTestUtils.randomBytes(fixedKeyLength), HashTableTestUtils.randomBytes(fixedValueLength), next);\n }\n\n @Test(expectedExceptions = IllegalArgumentException.class, expectedExceptionsMessageRegExp = \"Invalid request. Key length.*\")\n public void testWithInvalidKey() {\n int chunkSize = 256;\n int fixedKeyLength = 32, fixedValueLength = 100;\n MemoryPoolAddress next = new MemoryPoolAddress((byte)-1, -1);\n chunk = MemoryPoolChunk.create(chunkSize, fixedKeyLength, fixedValueLength);\n chunk.fillSlot(chunkSize - 5, HashTableTestUtils.randomBytes(fixedKeyLength + 10), HashTableTestUtils.randomBytes(fixedValueLength), next);\n }\n\n @Test\n public void testCompare() {\n int chunkSize = 1024;\n int fixedKeyLength = 9, fixedValueLength = 15;\n\n chunk = MemoryPoolChunk.create(chunkSize, fixedKeyLength, fixedValueLength);\n byte[] key = HashTableTestUtils.randomBytes(fixedKeyLength);\n byte[] value = HashTableTestUtils.randomBytes(fixedValueLength);\n int offset = 0;\n chunk.fillSlot(offset, key, value, new MemoryPoolAddress((byte)-1, -1));\n\n Assert.assertTrue(chunk.compareKey(offset, key));\n Assert.assertTrue(chunk.compareValue(offset, value));\n\n byte[] smallKey = new byte[key.length-1];\n System.arraycopy(key, 0, smallKey, 0, smallKey.length);\n Assert.assertFalse(chunk.compareKey(offset, smallKey));\n\n key[fixedKeyLength-1] = (byte)~key[fixedKeyLength-1];\n Assert.assertFalse(chunk.compareKey(offset, key));\n\n value[0] = (byte)~value[0];\n Assert.assertFalse(chunk.compareValue(offset, value));\n }\n\n @Test(expectedExceptions = IllegalArgumentException.class, expectedExceptionsMessageRegExp = \"Invalid request.*\")\n public void testCompareKeyWithException() {\n int chunkSize = 1024;\n Random r = new Random();\n int fixedKeyLength = 1 + r.nextInt(100), fixedValueLength = 1 + r.nextInt(100);\n \n chunk = MemoryPoolChunk.create(chunkSize, fixedKeyLength, fixedValueLength);\n byte[] key = HashTableTestUtils.randomBytes(fixedKeyLength);\n byte[] value = HashTableTestUtils.randomBytes(fixedValueLength);\n int offset = 0;\n chunk.fillSlot(offset, key, value, new MemoryPoolAddress((byte)-1, -1));\n\n byte[] bigKey = HashTableTestUtils.randomBytes(fixedKeyLength + 1);\n chunk.compareKey(offset, bigKey);\n\n\n }\n\n @Test(expectedExceptions = IllegalArgumentException.class, expectedExceptionsMessageRegExp = \"Invalid request.*\")\n public void testCompareValueWithException() {\n int chunkSize = 1024;\n Random r = new Random();\n int fixedKeyLength = 1 + r.nextInt(100), fixedValueLength = 1 + r.nextInt(100);\n\n chunk = MemoryPoolChunk.create(chunkSize, fixedKeyLength, fixedValueLength);\n byte[] key = HashTableTestUtils.randomBytes(fixedKeyLength);\n byte[] value = HashTableTestUtils.randomBytes(fixedValueLength);\n int offset = 0;\n chunk.fillSlot(offset, key, value, new MemoryPoolAddress((byte)-1, -1));\n\n byte[] bigValue = HashTableTestUtils.randomBytes(fixedValueLength + 1);\n chunk.compareValue(offset, bigValue);\n }\n\n @Test\n public void setAndGetNextAddress() {\n int chunkSize = 1024;\n Random r = new Random();\n int fixedKeyLength = 1 + r.nextInt(100), fixedValueLength = 1 + r.nextInt(100);\n\n chunk = MemoryPoolChunk.create(chunkSize, fixedKeyLength, fixedValueLength);\n\n MemoryPoolAddress nextAddress = new MemoryPoolAddress((byte)r.nextInt(Byte.MAX_VALUE), r.nextInt());\n int offset = r.nextInt(chunkSize - fixedKeyLength - fixedValueLength - MemoryPoolHashEntries.HEADER_SIZE);\n chunk.setNextAddress(offset, nextAddress);\n\n Assert.assertEquals(chunk.getNextAddress(offset), nextAddress);\n\n }\n}\n" }, { "path": "src/test/java/com/oath/halodb/NonMemoryPoolHashEntriesTest.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\n// This code is a derivative work heavily modified from the OHC project. See NOTICE file for copyright and license.\n\npackage com.oath.halodb;\n\nimport org.testng.annotations.AfterMethod;\nimport org.testng.annotations.Test;\n\nimport java.nio.ByteBuffer;\n\nimport static org.testng.Assert.*;\n\npublic class NonMemoryPoolHashEntriesTest\n{\n @AfterMethod(alwaysRun = true)\n public void deinit()\n {\n Uns.clearUnsDebugForTest();\n }\n\n static final long MIN_ALLOC_LEN = 128;\n\n @Test\n public void testInit() throws Exception\n {\n long adr = Uns.allocate(MIN_ALLOC_LEN);\n try\n {\n NonMemoryPoolHashEntries.init(5, adr);\n\n assertEquals(Uns.getLong(adr, NonMemoryPoolHashEntries.ENTRY_OFF_NEXT), 0L);\n assertEquals(Uns.getByte(adr, NonMemoryPoolHashEntries.ENTRY_OFF_KEY_LENGTH), 5);\n\n assertEquals(NonMemoryPoolHashEntries.getNext(adr), 0L);\n assertEquals(NonMemoryPoolHashEntries.getKeyLen(adr), 5L);\n }\n finally\n {\n Uns.free(adr);\n }\n }\n\n @Test\n public void testCompareKey() throws Exception\n {\n long adr = Uns.allocate(MIN_ALLOC_LEN);\n try\n {\n NonMemoryPoolHashEntries.init(11, adr);\n\n ByteBuffer buffer = ByteBuffer.allocate(11);\n buffer.putInt(0x98765432);\n buffer.putInt(0xabcdabba);\n buffer.put((byte)(0x44 & 0xff));\n buffer.put((byte)(0x55 & 0xff));\n buffer.put((byte)(0x88 & 0xff));\n\n KeyBuffer key = new KeyBuffer(buffer.array());\n key.finish(Hasher.create(HashAlgorithm.MURMUR3));\n\n Uns.setMemory(adr, NonMemoryPoolHashEntries.ENTRY_OFF_DATA, 11, (byte) 0);\n\n assertFalse(key.sameKey(adr));\n\n Uns.copyMemory(key.buffer, 0, adr, NonMemoryPoolHashEntries.ENTRY_OFF_DATA, 11);\n NonMemoryPoolHashEntries.init(11, adr);\n\n assertTrue(key.sameKey(adr));\n }\n finally\n {\n Uns.free(adr);\n }\n }\n\n @Test\n public void testGetSetNext() throws Exception\n {\n long adr = Uns.allocate(MIN_ALLOC_LEN);\n try\n {\n Uns.setMemory(adr, 0, MIN_ALLOC_LEN, (byte) 0);\n NonMemoryPoolHashEntries.init(5, adr);\n\n Uns.putLong(adr, NonMemoryPoolHashEntries.ENTRY_OFF_NEXT, 0x98765432abdffeedL);\n assertEquals(NonMemoryPoolHashEntries.getNext(adr), 0x98765432abdffeedL);\n\n NonMemoryPoolHashEntries.setNext(adr, 0xfafefcfb23242526L);\n assertEquals(Uns.getLong(adr, NonMemoryPoolHashEntries.ENTRY_OFF_NEXT), 0xfafefcfb23242526L);\n }\n finally\n {\n Uns.free(adr);\n }\n }\n}\n" }, { "path": "src/test/java/com/oath/halodb/OffHeapHashTableBuilderTest.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\n// This code is a derivative work heavily modified from the OHC project. See NOTICE file for copyright and license.\n\npackage com.oath.halodb;\n\nimport org.testng.Assert;\nimport org.testng.annotations.AfterMethod;\nimport org.testng.annotations.Test;\n\nimport java.nio.ByteBuffer;\nimport java.util.HashSet;\n\npublic class OffHeapHashTableBuilderTest\n{\n \n @Test\n public void testHashTableSize() throws Exception\n {\n OffHeapHashTableBuilder builder = OffHeapHashTableBuilder.newBuilder();\n Assert.assertEquals(builder.getHashTableSize(), 8192);\n builder.hashTableSize(12345);\n Assert.assertEquals(builder.getHashTableSize(), 12345);\n }\n\n @Test\n public void testChunkSize() throws Exception\n {\n OffHeapHashTableBuilder builder = OffHeapHashTableBuilder.newBuilder();\n builder.memoryPoolChunkSize(12345);\n Assert.assertEquals(builder.getMemoryPoolChunkSize(), 12345);\n }\n\n @Test\n public void testSegmentCount() throws Exception\n {\n int cpus = Runtime.getRuntime().availableProcessors();\n int segments = cpus * 2;\n while (Integer.bitCount(segments) != 1)\n segments++;\n\n OffHeapHashTableBuilder builder = OffHeapHashTableBuilder.newBuilder();\n Assert.assertEquals(builder.getSegmentCount(), segments);\n builder.segmentCount(12345);\n Assert.assertEquals(builder.getSegmentCount(), 12345);\n }\n\n @Test\n public void testLoadFactor() throws Exception\n {\n OffHeapHashTableBuilder builder = OffHeapHashTableBuilder.newBuilder();\n Assert.assertEquals(builder.getLoadFactor(), .75f);\n builder.loadFactor(12345);\n Assert.assertEquals(builder.getLoadFactor(), 12345.0f);\n }\n\n @Test\n public void testValueSerializer() throws Exception\n {\n OffHeapHashTableBuilder builder = OffHeapHashTableBuilder.newBuilder();\n Assert.assertNull(builder.getValueSerializer());\n\n HashTableValueSerializer inst = new HashTableValueSerializer()\n {\n public void serialize(String s, ByteBuffer out)\n {\n\n }\n\n public String deserialize(ByteBuffer in)\n {\n return null;\n }\n\n public int serializedSize(String s)\n {\n return 0;\n }\n };\n builder.valueSerializer(inst);\n Assert.assertSame(builder.getValueSerializer(), inst);\n }\n\n @Test(expectedExceptions = IllegalArgumentException.class, expectedExceptionsMessageRegExp = \".*Need to set fixedValueSize*\")\n public void testFixedValueSize() throws Exception {\n OffHeapHashTableBuilder builder = OffHeapHashTableBuilder.newBuilder();\n builder.build();\n }\n}\n" }, { "path": "src/test/java/com/oath/halodb/RandomDataGenerator.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport java.util.Random;\n\nclass RandomDataGenerator {\n\n private final byte[] data;\n private static final int size = 1003087;\n private int position = 0;\n\n RandomDataGenerator() {\n this.data = new byte[size];\n Random random = new Random();\n random.nextBytes(data);\n }\n\n byte[] getData(int length) {\n byte[] b = new byte[length];\n\n for (int i = 0; i < length; i++) {\n if (position >= size) {\n position = 0;\n }\n\n b[i] = data[position++];\n }\n\n return b;\n }\n}" }, { "path": "src/test/java/com/oath/halodb/RecordTest.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport org.testng.Assert;\nimport org.testng.annotations.Test;\n\nimport java.nio.ByteBuffer;\nimport java.util.zip.CRC32;\n\npublic class RecordTest {\n\n @Test\n public void testSerializeHeader() {\n\n byte keySize = 8;\n int valueSize = 100;\n long sequenceNumber = 34543434343L;\n int version = 128;\n\n Record.Header header = new Record.Header(0, version, keySize, valueSize, sequenceNumber);\n ByteBuffer serialized = header.serialize();\n\n Assert.assertEquals(keySize, serialized.get(Record.Header.KEY_SIZE_OFFSET));\n Assert.assertEquals(valueSize, serialized.getInt(Record.Header.VALUE_SIZE_OFFSET));\n Assert.assertEquals(sequenceNumber, serialized.getLong(Record.Header.SEQUENCE_NUMBER_OFFSET));\n Assert.assertEquals(Utils.toUnsignedByte(serialized.get(Record.Header.VERSION_OFFSET)), version);\n }\n\n @Test\n public void testDeserializeHeader() {\n\n long checkSum = 23434;\n byte keySize = 8;\n int valueSize = 100;\n long sequenceNumber = 34543434343L;\n int version = 2;\n\n ByteBuffer buffer = ByteBuffer.allocate(Record.Header.HEADER_SIZE);\n buffer.putInt(Utils.toSignedIntFromLong(checkSum));\n buffer.put((byte)version);\n buffer.put(keySize);\n buffer.putInt(valueSize);\n buffer.putLong(sequenceNumber);\n buffer.flip();\n\n Record.Header header = Record.Header.deserialize(buffer);\n\n Assert.assertEquals(checkSum, header.getCheckSum());\n Assert.assertEquals(version, header.getVersion());\n Assert.assertEquals(keySize, header.getKeySize());\n Assert.assertEquals(valueSize, header.getValueSize());\n Assert.assertEquals(sequenceNumber, header.getSequenceNumber());\n Assert.assertEquals(keySize + valueSize + Record.Header.HEADER_SIZE, header.getRecordSize());\n }\n\n @Test\n public void testSerializeRecord() {\n byte[] key = TestUtils.generateRandomByteArray();\n byte[] value = TestUtils.generateRandomByteArray();\n long sequenceNumber = 192;\n int version = 13;\n\n Record record = new Record(key, value);\n record.setSequenceNumber(sequenceNumber);\n record.setVersion(version);\n\n ByteBuffer[] buffers = record.serialize();\n CRC32 crc32 = new CRC32();\n crc32.update(buffers[0].array(), Record.Header.VERSION_OFFSET, buffers[0].array().length - Record.Header.CHECKSUM_SIZE);\n crc32.update(key);\n crc32.update(value);\n\n Record.Header header = new Record.Header(0, version, (byte)key.length, value.length, sequenceNumber);\n ByteBuffer headerBuf = header.serialize();\n headerBuf.putInt(Record.Header.CHECKSUM_OFFSET, Utils.toSignedIntFromLong(crc32.getValue()));\n\n Assert.assertEquals(headerBuf, buffers[0]);\n Assert.assertEquals(ByteBuffer.wrap(key), buffers[1]);\n Assert.assertEquals(ByteBuffer.wrap(value), buffers[2]);\n }\n}\n" }, { "path": "src/test/java/com/oath/halodb/RehashTest.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\n// This code is a derivative work heavily modified from the OHC project. See NOTICE file for copyright and license.\n\npackage com.oath.halodb;\n\nimport com.google.common.primitives.Longs;\n\nimport org.testng.annotations.AfterMethod;\nimport org.testng.annotations.Test;\n\nimport java.io.IOException;\n\nimport static org.testng.Assert.assertEquals;\nimport static org.testng.Assert.assertTrue;\n\npublic class RehashTest\n{\n @AfterMethod(alwaysRun = true)\n public void deinit()\n {\n Uns.clearUnsDebugForTest();\n }\n\n @Test\n public void testRehash() throws IOException\n {\n try (OffHeapHashTable cache = OffHeapHashTableBuilder.newBuilder()\n .valueSerializer(HashTableTestUtils.byteArraySerializer)\n .hashTableSize(64)\n .segmentCount(4)\n .fixedValueSize(8)\n .build())\n {\n for (int i = 0; i < 100000; i++)\n cache.put(Longs.toByteArray(i), Longs.toByteArray(i));\n\n assertTrue(cache.stats().getRehashCount() > 0);\n\n for (int i = 0; i < 100000; i++)\n {\n byte[] v = cache.get(Longs.toByteArray(i));\n assertEquals(Longs.fromByteArray(v), i);\n }\n }\n }\n}\n" }, { "path": "src/test/java/com/oath/halodb/SegmentWithMemoryPoolTest.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport com.google.common.collect.Lists;\n\nimport org.testng.Assert;\nimport org.testng.annotations.Test;\n\nimport java.util.ArrayList;\nimport java.util.HashMap;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.Random;\n\npublic class SegmentWithMemoryPoolTest {\n\n @Test\n public void testChunkAllocations() {\n\n int fixedKeySize = 8;\n int fixedValueSize = 18;\n int noOfEntries = 100;\n int noOfChunks = 2;\n int fixedSlotSize = MemoryPoolHashEntries.HEADER_SIZE + fixedKeySize + fixedValueSize;\n\n OffHeapHashTableBuilder builder = OffHeapHashTableBuilder\n .newBuilder()\n .fixedKeySize(fixedKeySize)\n .fixedValueSize(fixedValueSize)\n // chunkSize set such that noOfEntries/2 can set filled in one. \n .memoryPoolChunkSize(noOfEntries/noOfChunks * fixedSlotSize)\n .valueSerializer(HashTableTestUtils.byteArraySerializer);\n\n SegmentWithMemoryPool segment = new SegmentWithMemoryPool<>(builder);\n\n addEntriesToSegment(segment, Hasher.create(HashAlgorithm.MURMUR3), noOfEntries, fixedKeySize, fixedValueSize);\n\n // each chunk can hold noOfEntries/2 hence two chunks must be allocated. \n Assert.assertEquals(segment.numberOfChunks(), 2);\n Assert.assertEquals(segment.numberOfSlots(), noOfEntries);\n Assert.assertEquals(segment.size(), noOfEntries);\n Assert.assertEquals(segment.putAddCount(), noOfEntries);\n Assert.assertEquals(segment.freeListSize(), 0);\n\n // All slots in chunk should be written to. \n for (int i = 0; i < segment.numberOfChunks(); i++) {\n Assert.assertEquals(segment.getChunkWriteOffset(i), noOfEntries/noOfChunks * fixedSlotSize);\n }\n }\n\n @Test\n public void testFreeList() {\n int fixedKeySize = 8;\n int fixedValueSize = 18;\n int noOfEntries = 100;\n int chunkCount = 2;\n int fixedSlotSize = MemoryPoolHashEntries.HEADER_SIZE + fixedKeySize + fixedValueSize;\n MemoryPoolAddress emptyList = new MemoryPoolAddress((byte) -1, -1);\n\n OffHeapHashTableBuilder builder = OffHeapHashTableBuilder\n .newBuilder()\n .fixedKeySize(fixedKeySize)\n .fixedValueSize(fixedValueSize)\n // chunkSize set such that noOfEntries/2 can set filled in one.\n .memoryPoolChunkSize(noOfEntries / chunkCount * fixedSlotSize)\n .valueSerializer(HashTableTestUtils.byteArraySerializer);\n\n SegmentWithMemoryPool segment = new SegmentWithMemoryPool<>(builder);\n\n //Add noOfEntries to the segment. This should require chunks.\n Hasher hasher = Hasher.create(HashAlgorithm.MURMUR3);\n List records = addEntriesToSegment(segment, hasher, noOfEntries, fixedKeySize, fixedValueSize);\n\n // each chunk can hold noOfEntries/2 hence two chunks must be allocated.\n Assert.assertEquals(segment.numberOfChunks(), chunkCount);\n Assert.assertEquals(segment.size(), noOfEntries);\n Assert.assertEquals(segment.putAddCount(), noOfEntries);\n Assert.assertEquals(segment.freeListSize(), 0);\n Assert.assertEquals(segment.getFreeListHead(), emptyList);\n\n // remove all entries from the segment\n // all slots should now be part of the free list. \n Lists.reverse(records).forEach(k -> segment.removeEntry(k.keyBuffer));\n\n Assert.assertEquals(segment.freeListSize(), noOfEntries);\n Assert.assertNotEquals(segment.getFreeListHead(), emptyList);\n Assert.assertEquals(segment.removeCount(), noOfEntries);\n Assert.assertEquals(segment.size(), 0);\n\n // Add noOfEntries to the segment.\n // All entries must be added to slots part of the freelist. \n records = addEntriesToSegment(segment, hasher, noOfEntries, fixedKeySize, fixedValueSize);\n\n Assert.assertEquals(segment.numberOfChunks(), chunkCount);\n Assert.assertEquals(segment.size(), noOfEntries);\n Assert.assertEquals(segment.freeListSize(), 0);\n\n // after all slots in the free list are used head should point to\n // an empty list. \n Assert.assertEquals(segment.getFreeListHead(), emptyList);\n\n // remove only some of the elements.\n Random r = new Random();\n int elementsRemoved = 0;\n for (int i = 0; i < noOfEntries/3; i++) {\n if(segment.removeEntry(records.get(r.nextInt(records.size())).keyBuffer))\n elementsRemoved++;\n }\n\n Assert.assertEquals(segment.freeListSize(), elementsRemoved);\n Assert.assertNotEquals(segment.getFreeListHead(), emptyList);\n Assert.assertEquals(segment.size(), noOfEntries-elementsRemoved);\n\n // add removed elements back.\n addEntriesToSegment(segment, hasher, elementsRemoved, fixedKeySize, fixedValueSize);\n\n Assert.assertEquals(segment.numberOfChunks(), chunkCount);\n Assert.assertEquals(segment.size(), noOfEntries);\n Assert.assertEquals(segment.freeListSize(), 0);\n Assert.assertEquals(segment.getFreeListHead(), emptyList);\n }\n\n @Test(expectedExceptions = OutOfMemoryError.class, expectedExceptionsMessageRegExp = \"Each segment can have at most 128 chunks.\")\n public void testOutOfMemoryException() {\n int fixedKeySize = 8;\n int fixedValueSize = 18;\n int fixedSlotSize = MemoryPoolHashEntries.HEADER_SIZE + fixedKeySize + fixedValueSize;\n\n // Each segment can have only Byte.MAX_VALUE chunks.\n // we add more that that.\n int noOfEntries = Byte.MAX_VALUE * 2;\n\n OffHeapHashTableBuilder builder = OffHeapHashTableBuilder\n .newBuilder()\n .fixedKeySize(fixedKeySize)\n .fixedValueSize(fixedValueSize)\n // chunkSize set so that each can contain only one entry.\n .memoryPoolChunkSize(fixedSlotSize)\n .valueSerializer(HashTableTestUtils.byteArraySerializer);\n\n SegmentWithMemoryPool segment = new SegmentWithMemoryPool<>(builder);\n addEntriesToSegment(segment, Hasher.create(HashAlgorithm.MURMUR3), noOfEntries, fixedKeySize, fixedValueSize);\n }\n\n @Test\n public void testReplace() {\n\n int fixedKeySize = 8;\n int fixedValueSize = 18;\n int noOfEntries = 1000;\n int noOfChunks = 10;\n int fixedSlotSize = MemoryPoolHashEntries.HEADER_SIZE + fixedKeySize + fixedValueSize;\n\n OffHeapHashTableBuilder builder = OffHeapHashTableBuilder\n .newBuilder()\n .fixedKeySize(fixedKeySize)\n .fixedValueSize(fixedValueSize)\n // chunkSize set such that noOfEntries/2 can set filled in one.\n .memoryPoolChunkSize(noOfEntries/noOfChunks * fixedSlotSize)\n .valueSerializer(HashTableTestUtils.byteArraySerializer);\n\n SegmentWithMemoryPool segment = new SegmentWithMemoryPool<>(builder);\n\n Hasher hasher = Hasher.create(HashAlgorithm.MURMUR3);\n Map map = new HashMap<>();\n for (int i = 0; i < noOfEntries; i++) {\n byte[] key = HashTableTestUtils.randomBytes(fixedKeySize);\n KeyBuffer k = new KeyBuffer(key);\n k.finish(hasher);\n byte[] value = HashTableTestUtils.randomBytes(fixedValueSize);\n map.put(k, value);\n segment.putEntry(key, value, k.hash(), true, null);\n }\n\n Assert.assertEquals(segment.numberOfChunks(), noOfChunks);\n Assert.assertEquals(segment.size(), noOfEntries);\n Assert.assertEquals(segment.putAddCount(), noOfEntries);\n Assert.assertEquals(segment.freeListSize(), 0);\n\n map.forEach((k, v) -> {\n Assert.assertTrue(segment.putEntry(k.buffer, HashTableTestUtils.randomBytes(fixedValueSize), k.hash(), false, v));\n });\n\n // we have replaced all values. no new chunks should\n // have been allocated.\n Assert.assertEquals(segment.numberOfChunks(), noOfChunks);\n Assert.assertEquals(segment.size(), noOfEntries);\n Assert.assertEquals(segment.putAddCount(), noOfEntries);\n Assert.assertEquals(segment.freeListSize(), 0);\n Assert.assertEquals(segment.putReplaceCount(), noOfEntries);\n\n // All slots in chunk should be written to.\n for (int i = 0; i < segment.numberOfChunks(); i++) {\n Assert.assertEquals(segment.getChunkWriteOffset(i), noOfEntries/noOfChunks * fixedSlotSize);\n }\n }\n\n @Test\n public void testRehash() {\n\n int fixedKeySize = 8;\n int fixedValueSize = 18;\n int noOfEntries = 100_000;\n int noOfChunks = 10;\n int fixedSlotSize = MemoryPoolHashEntries.HEADER_SIZE + fixedKeySize + fixedValueSize;\n\n OffHeapHashTableBuilder builder = OffHeapHashTableBuilder\n .newBuilder()\n .fixedKeySize(fixedKeySize)\n .fixedValueSize(fixedValueSize)\n .memoryPoolChunkSize(noOfEntries/noOfChunks * fixedSlotSize)\n .hashTableSize(noOfEntries/8) // size of table less than number of entries, this will trigger a rehash.\n .loadFactor(1)\n .valueSerializer(HashTableTestUtils.byteArraySerializer);\n\n SegmentWithMemoryPool segment = new SegmentWithMemoryPool<>(builder);\n Hasher hasher = Hasher.create(HashAlgorithm.MURMUR3);\n List records = addEntriesToSegment(segment, hasher, noOfEntries, fixedKeySize, fixedValueSize);\n\n Assert.assertEquals(segment.size(), noOfEntries);\n Assert.assertEquals(segment.rehashes(), 3);\n Assert.assertEquals(segment.putAddCount(), noOfEntries);\n\n records.forEach(r -> Assert.assertEquals(segment.getEntry(r.keyBuffer), r.value));\n }\n\n\n\n private List addEntriesToSegment(SegmentWithMemoryPool segment, Hasher hasher, int noOfEntries, int fixedKeySize, int fixedValueSize) {\n List records = new ArrayList<>();\n for (int i = 0; i < noOfEntries; i++) {\n byte[] key = HashTableTestUtils.randomBytes(fixedKeySize);\n KeyBuffer k = new KeyBuffer(key);\n k.finish(hasher);\n byte[] value = HashTableTestUtils.randomBytes(fixedValueSize);\n records.add(new Record(k, value));\n segment.putEntry(key, value, k.hash(), true, null);\n }\n\n return records;\n }\n\n private static class Record {\n final KeyBuffer keyBuffer;\n final byte[] value;\n\n public Record(KeyBuffer keyBuffer, byte[] value) {\n this.keyBuffer = keyBuffer;\n this.value = value;\n }\n }\n}\n" }, { "path": "src/test/java/com/oath/halodb/SequenceNumberTest.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport org.testng.Assert;\nimport org.testng.annotations.Test;\n\nimport java.io.File;\nimport java.util.ArrayList;\nimport java.util.Arrays;\nimport java.util.Comparator;\nimport java.util.List;\nimport java.util.stream.Collectors;\n\n/**\n * @author Pulkit Goel\n */\npublic class SequenceNumberTest extends TestBase {\n\n @Test(dataProvider = \"Options\")\n public void testSequenceNumber(HaloDBOptions options) throws Exception {\n String directory = TestUtils.getTestDirectory(\"SequenceNumberTest\", \"testSequenceNumber\");\n\n int totalNumberOfRecords = 10;\n options.setMaxFileSize(1024 * 1024 * 1024);\n\n // Write 10 records in the DB\n HaloDB db = getTestDB(directory, options);\n TestUtils.insertRandomRecords(db, totalNumberOfRecords);\n\n // Iterate through all records, atleast one record should have 10 as the sequenceNumber\n File file = Arrays.stream(FileUtils.listDataFiles(new File(directory))).max(Comparator.comparing(File::getName)).get();\n HaloDBFile.HaloDBFileIterator haloDBFileIterator = HaloDBFile.openForReading(dbDirectory, file, HaloDBFile.FileType.DATA_FILE, options).newIterator();\n\n List sequenceNumbers = new ArrayList<>();\n int count = 1;\n while (haloDBFileIterator.hasNext()) {\n Record record = haloDBFileIterator.next();\n sequenceNumbers.add(record.getSequenceNumber());\n Assert.assertEquals(record.getSequenceNumber(), count++);\n }\n Assert.assertTrue(sequenceNumbers.contains(10L));\n db.close();\n\n // open and read the content again\n HaloDB reopenedDb = getTestDBWithoutDeletingFiles(directory, options);\n List records = new ArrayList<>();\n reopenedDb.newIterator().forEachRemaining(records::add);\n\n // Verify that the sequence number is still present after reopening the DB\n sequenceNumbers = records.stream().map(record -> record.getRecordMetaData().getSequenceNumber()).collect(Collectors.toList());\n count = 1;\n for (long s : sequenceNumbers) {\n Assert.assertEquals(s, count++);\n }\n Assert.assertTrue(sequenceNumbers.contains(10L));\n\n // Write 10 records in the DB\n TestUtils.insertRandomRecords(reopenedDb, totalNumberOfRecords);\n\n // Iterate through all records, atleast one record should have 119 as the sequenceNumber (10 original records + 99 offset for reopening + 10 new records)\n file = Arrays.stream(FileUtils.listDataFiles(new File(directory))).max(Comparator.comparing(File::getName)).get();\n haloDBFileIterator = HaloDBFile.openForReading(dbDirectory, file, HaloDBFile.FileType.DATA_FILE, options).newIterator();\n\n sequenceNumbers = new ArrayList<>();\n count = 110;\n while (haloDBFileIterator.hasNext()) {\n Record record = haloDBFileIterator.next();\n sequenceNumbers.add(record.getSequenceNumber());\n Assert.assertEquals(record.getSequenceNumber(), count++);\n }\n Assert.assertTrue(sequenceNumbers.contains(119L));\n\n // Delete the first 10 records from the DB\n for (Record record : records) {\n reopenedDb.delete(record.getKey());\n }\n\n // get the tombstone file.\n File[] tombstoneFiles = FileUtils.listTombstoneFiles(new File(directory));\n Assert.assertEquals(tombstoneFiles.length, 1);\n\n TombstoneFile tombstoneFile = new TombstoneFile(tombstoneFiles[0], options, dbDirectory);\n tombstoneFile.open();\n List tombstoneEntries = new ArrayList<>();\n tombstoneFile.newIterator().forEachRemaining(tombstoneEntries::add);\n\n Assert.assertEquals(tombstoneEntries.size(), 10);\n count = 120;\n for (TombstoneEntry tombstoneEntry : tombstoneEntries) {\n // Each tombstoneEntry should have sequence number greater than or equal to 119 (10 original records + 99 offset for reopening + 10 new records)\n Assert.assertEquals(tombstoneEntry.getSequenceNumber(), count++);\n }\n reopenedDb.close();\n\n // reopen the db and add the content again\n reopenedDb = getTestDBWithoutDeletingFiles(directory, options);\n\n // Write 10 records in the DB\n TestUtils.insertRandomRecords(reopenedDb, totalNumberOfRecords);\n\n // Iterate through all records, atleast one record should have 238 as the sequenceNumber (10 original records + 99 offset for reopening + 10 records + 10 tombstone records + 99 offset for reopening + 10 new records)\n file = Arrays.stream(FileUtils.listDataFiles(new File(directory))).max(Comparator.comparing(File::getName)).get();\n haloDBFileIterator = HaloDBFile.openForReading(dbDirectory, file, HaloDBFile.FileType.DATA_FILE, options).newIterator();\n\n sequenceNumbers = new ArrayList<>();\n count = 229;\n while (haloDBFileIterator.hasNext()) {\n Record record = haloDBFileIterator.next();\n sequenceNumbers.add(record.getSequenceNumber());\n Assert.assertEquals(record.getSequenceNumber(), count++);\n }\n Assert.assertTrue(sequenceNumbers.contains(238L));\n reopenedDb.close();\n }\n}\n" }, { "path": "src/test/java/com/oath/halodb/SyncWriteTest.java", "content": "package com.oath.halodb;\n\nimport org.testng.Assert;\nimport org.testng.annotations.Test;\n\nimport java.io.IOException;\nimport java.util.List;\nimport java.util.concurrent.atomic.AtomicInteger;\n\nimport mockit.Invocation;\nimport mockit.Mock;\nimport mockit.MockUp;\n\npublic class SyncWriteTest extends TestBase {\n\n @Test\n public void testSyncWrites() throws HaloDBException {\n String directory = TestUtils.getTestDirectory(\"SyncWriteTest\", \"testSyncWrites\");\n\n AtomicInteger dataFileCount = new AtomicInteger(0);\n AtomicInteger tombstoneFileCount = new AtomicInteger(0);\n\n new MockUp() {\n @Mock\n public void flushToDisk(Invocation invocation) throws IOException {\n dataFileCount.incrementAndGet();\n }\n };\n\n new MockUp() {\n @Mock\n public void flushToDisk(Invocation invocation) throws IOException {\n tombstoneFileCount.incrementAndGet();\n }\n };\n\n HaloDBOptions options = new HaloDBOptions();\n options.enableSyncWrites(true);\n\n HaloDB db = getTestDB(directory, options);\n int noOfRecords = 10;\n List records = TestUtils.insertRandomRecords(db, noOfRecords);\n for (Record r : records) {\n db.delete(r.getKey());\n }\n\n // since sync write is enabled each record should have been flushed after write.\n Assert.assertEquals(dataFileCount.get(), noOfRecords);\n Assert.assertEquals(tombstoneFileCount.get(), noOfRecords);\n }\n\n @Test\n public void testNonSyncWrites() throws HaloDBException {\n String directory = TestUtils.getTestDirectory(\"SyncWriteTest\", \"testNonSyncWrites\");\n\n AtomicInteger dataFileCount = new AtomicInteger(0);\n new MockUp() {\n @Mock\n public void flushToDisk(Invocation invocation) throws IOException {\n dataFileCount.incrementAndGet();\n }\n };\n\n HaloDBOptions options = new HaloDBOptions();\n // value set to make sure that flush to disk will be called once. \n options.setFlushDataSizeBytes(10 * 1024 - 1);\n\n HaloDB db = getTestDB(directory, options);\n int noOfRecords = 10;\n TestUtils.insertRandomRecordsOfSize(db, noOfRecords, 1024 - Record.Header.HEADER_SIZE);\n\n // 10 records of size 1024 each was inserted and flush size was set to 10 * 1024 - 1,\n // therefore data will be flushed to disk once. \n Assert.assertEquals(dataFileCount.get(), 1);\n }\n\n @Test\n public void testNonSyncDeletes() throws HaloDBException {\n String directory = TestUtils.getTestDirectory(\"SyncWriteTest\", \"testNonSyncDeletes\");\n\n AtomicInteger dataFileCount = new AtomicInteger(0);\n AtomicInteger tombstoneFileCount = new AtomicInteger(0);\n new MockUp() {\n @Mock\n public void flushToDisk(Invocation invocation) throws IOException {\n dataFileCount.incrementAndGet();\n }\n };\n\n new MockUp() {\n @Mock\n public void flushToDisk(Invocation invocation) throws IOException {\n tombstoneFileCount.incrementAndGet();\n }\n };\n\n HaloDBOptions options = new HaloDBOptions();\n // value set to make sure that flush to disk will not be called. \n options.setFlushDataSizeBytes(1024 * 1024 * 1024);\n\n HaloDB db = getTestDB(directory, options);\n int noOfRecords = 100;\n List records = TestUtils.insertRandomRecordsOfSize(db, noOfRecords, 1024 - Record.Header.HEADER_SIZE);\n for (Record r : records) {\n db.delete(r.getKey());\n }\n\n // each record should have been flushed after write.\n Assert.assertEquals(dataFileCount.get(), 0);\n Assert.assertEquals(tombstoneFileCount.get(), 0);\n }\n\n}\n" }, { "path": "src/test/java/com/oath/halodb/TestBase.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport org.testng.annotations.AfterMethod;\nimport org.testng.annotations.DataProvider;\n\nimport java.io.File;\nimport java.io.IOException;\n\npublic class TestBase {\n\n private String directory;\n protected DBDirectory dbDirectory;\n\n private HaloDB db;\n\n @DataProvider(name = \"Options\")\n public Object[][] optionData() {\n HaloDBOptions options = new HaloDBOptions();\n options.setBuildIndexThreads(2);\n HaloDBOptions withMemoryPool = new HaloDBOptions();\n withMemoryPool.setUseMemoryPool(true);\n withMemoryPool.setMemoryPoolChunkSize(1024 * 1024);\n withMemoryPool.setBuildIndexThreads(2);\n\n return new Object[][] {\n {options},\n {withMemoryPool}\n };\n }\n\n HaloDB getTestDB(String directory, HaloDBOptions options) throws HaloDBException {\n this.directory = directory;\n File dir = new File(directory);\n try {\n TestUtils.deleteDirectory(dir);\n } catch (IOException e) {\n throw new HaloDBException(e);\n }\n db = HaloDB.open(dir, options);\n try {\n dbDirectory = DBDirectory.open(new File(directory));\n } catch (IOException e) {\n throw new HaloDBException(e);\n }\n return db;\n }\n\n HaloDB getTestDBWithoutDeletingFiles(String directory, HaloDBOptions options) throws HaloDBException {\n this.directory = directory;\n File dir = new File(directory);\n db = HaloDB.open(dir, options);\n TestUtils.waitForTombstoneFileMergeComplete(db);\n return db;\n }\n\n @AfterMethod(alwaysRun = true)\n public void closeDB() throws HaloDBException, IOException {\n if (db != null) {\n db.close();\n db = null;\n File dir = new File(directory);\n if (dbDirectory != null) {\n dbDirectory.close();\n dbDirectory = null;\n }\n TestUtils.deleteDirectory(dir);\n }\n }\n}\n" }, { "path": "src/test/java/com/oath/halodb/TestListener.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\nimport org.testng.ITestContext;\nimport org.testng.ITestListener;\nimport org.testng.ITestResult;\n\npublic class TestListener implements ITestListener {\n private static final Logger logger = LoggerFactory.getLogger(TestListener.class);\n\n @Override\n public void onTestStart(ITestResult result) {\n logger.info(\"Running {}.{}\", result.getTestClass().getName(), result.getMethod().getMethodName());\n\n }\n\n @Override\n public void onTestSuccess(ITestResult result) {\n logger.info(\"Success {}.{}\", result.getTestClass().getName(), result.getMethod().getMethodName());\n\n }\n\n @Override\n public void onTestFailure(ITestResult result) {\n logger.info(\"Failure {}.{}\", result.getTestClass().getName(), result.getMethod().getMethodName());\n }\n\n @Override\n public void onTestSkipped(ITestResult result) {\n\n }\n\n @Override\n public void onTestFailedButWithinSuccessPercentage(ITestResult result) {\n\n }\n\n @Override\n public void onStart(ITestContext context) {\n\n }\n\n @Override\n public void onFinish(ITestContext context) {\n\n }\n}\n" }, { "path": "src/test/java/com/oath/halodb/TestUtils.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport java.io.File;\nimport java.io.IOException;\nimport java.nio.ByteBuffer;\nimport java.nio.file.FileVisitResult;\nimport java.nio.file.Files;\nimport java.nio.file.Path;\nimport java.nio.file.Paths;\nimport java.nio.file.SimpleFileVisitor;\nimport java.nio.file.attribute.BasicFileAttributes;\nimport java.nio.file.attribute.FileTime;\nimport java.util.ArrayList;\nimport java.util.Arrays;\nimport java.util.Comparator;\nimport java.util.HashSet;\nimport java.util.List;\nimport java.util.Optional;\nimport java.util.Random;\nimport java.util.Set;\nimport java.util.stream.Collectors;\n\npublic class TestUtils {\n private static final Logger logger = LoggerFactory.getLogger(TestUtils.class);\n\n static String getTestDirectory(String... path) {\n return getTestDirectoryPath(path).toString();\n }\n\n static Path getTestDirectoryPath(String... path) {\n return Paths.get(\"tmp\", path);\n }\n\n static List insertRandomRecords(HaloDB db, int noOfRecords) throws HaloDBException {\n return insertRandomRecordsOfSize(db, noOfRecords, -1);\n }\n\n static List insertRandomRecordsOfSize(HaloDB db, int noOfRecords, int size) throws HaloDBException {\n List records = new ArrayList<>();\n Set keySet = new HashSet<>();\n Random random = new Random();\n\n for (int i = 0; i < noOfRecords; i++) {\n byte[] key;\n if (size > 0) {\n key = TestUtils.generateRandomByteArray(random.nextInt(Math.min(Byte.MAX_VALUE-1, size))+1);\n }\n else {\n key = TestUtils.generateRandomByteArray();\n }\n while (keySet.contains(ByteBuffer.wrap(key))) {\n key = TestUtils.generateRandomByteArray();\n }\n ByteBuffer buf = ByteBuffer.wrap(key);\n keySet.add(buf);\n\n byte[] value;\n if (size > 0) {\n value = TestUtils.generateRandomByteArray(size - key.length);\n }\n else {\n value = TestUtils.generateRandomByteArray();\n }\n records.add(new Record(key, value));\n\n db.put(key, value);\n }\n\n return records;\n }\n\n static List generateRandomData(int noOfRecords) {\n List records = new ArrayList<>();\n Set keySet = new HashSet<>();\n\n for (int i = 0; i < noOfRecords; i++) {\n byte[] key = TestUtils.generateRandomByteArray();\n while (keySet.contains(ByteBuffer.wrap(key))) {\n key = TestUtils.generateRandomByteArray();\n }\n ByteBuffer buf = ByteBuffer.wrap(key);\n keySet.add(buf);\n\n byte[] value = TestUtils.generateRandomByteArray();\n records.add(new Record(key, value));\n }\n\n return records;\n\n }\n\n static List updateRecords(HaloDB db, List records) {\n List updated = new ArrayList<>();\n\n records.forEach(record -> {\n try {\n byte[] value = TestUtils.generateRandomByteArray();\n db.put(record.getKey(), value);\n updated.add(new Record(record.getKey(), value));\n } catch (HaloDBException e) {\n throw new RuntimeException(e);\n }\n });\n\n return updated;\n }\n\n static List updateRecordsWithSize(HaloDB db, List records, int size) {\n List updated = new ArrayList<>();\n\n records.forEach(record -> {\n try {\n byte[] value = TestUtils.generateRandomByteArray(size-record.getKey().length-Record.Header.HEADER_SIZE);\n db.put(record.getKey(), value);\n updated.add(new Record(record.getKey(), value));\n } catch (HaloDBException e) {\n throw new RuntimeException(e);\n }\n });\n\n return updated;\n }\n\n static void deleteRecords(HaloDB db, List records) {\n records.forEach(r -> {\n try {\n db.delete(r.getKey());\n } catch (HaloDBException e) {\n throw new RuntimeException(e);\n }\n });\n }\n\n static void deleteDirectory(File directory) throws IOException {\n if (!directory.exists())\n return;\n\n Path path = Paths.get(directory.getPath());\n SimpleFileVisitor visitor = new SimpleFileVisitor() {\n\n @Override\n public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) throws IOException {\n Files.delete(file);\n return FileVisitResult.CONTINUE;\n }\n\n @Override\n public FileVisitResult postVisitDirectory(Path dir, IOException exc) throws IOException {\n Files.delete(dir);\n return FileVisitResult.CONTINUE;\n }\n };\n\n Files.walkFileTree(path, visitor);\n }\n\n static byte[] concatenateArrays(byte[] a, byte[] b) {\n byte[] c = new byte[a.length + b.length];\n System.arraycopy(a, 0, c, 0, a.length);\n System.arraycopy(b, 0, c, a.length, b.length);\n\n return c;\n }\n\n private static Random random = new Random();\n\n static String generateRandomAsciiString(int length) {\n StringBuilder builder = new StringBuilder(length);\n\n for (int i = 0; i < length; i++) {\n int next = 48 + random.nextInt(78);\n builder.append((char)next);\n }\n\n return builder.toString();\n }\n\n static String generateRandomAsciiString() {\n int length = random.nextInt(20) + 1;\n StringBuilder builder = new StringBuilder(length);\n\n for (int i = 0; i < length; i++) {\n int next = 48 + random.nextInt(78);\n builder.append((char)next);\n }\n\n return builder.toString();\n }\n\n public static byte[] generateRandomByteArray(int length) {\n byte[] array = new byte[length];\n random.nextBytes(array);\n\n return array;\n }\n\n public static byte[] generateRandomByteArray() {\n int length = random.nextInt(Byte.MAX_VALUE) + 1;\n byte[] array = new byte[length];\n random.nextBytes(array);\n\n return array;\n }\n\n /**\n * This method will work correctly only after all the writes to the db have been completed.\n */\n static void waitForCompactionToComplete(HaloDB db) {\n while (!db.isCompactionComplete()) {\n try {\n Thread.sleep(1_000);\n } catch (InterruptedException e) {\n logger.error(\"Thread interrupted while waiting for compaction to complete\");\n throw new RuntimeException(e);\n }\n }\n }\n\n static void waitForTombstoneFileMergeComplete(HaloDB db) {\n while (db.isTombstoneFilesMerging()) {\n try {\n Thread.sleep(1_000);\n } catch (InterruptedException e) {\n logger.error(\"Thread interrupted while waiting for tombstone file merge to complete\");\n throw new RuntimeException(e);\n }\n }\n }\n\n static Optional getLatestDataFile(String directory) {\n return Arrays.stream(FileUtils.listDataFiles(new File(directory)))\n .filter(f -> HaloDBFile.findFileType(f) == HaloDBFile.FileType.DATA_FILE)\n .max(Comparator.comparing(File::getName));\n }\n\n static List getDataFiles(String directory) {\n return Arrays.stream(FileUtils.listDataFiles(new File(directory)))\n .filter(f -> HaloDBFile.findFileType(f) == HaloDBFile.FileType.DATA_FILE)\n .collect(Collectors.toList());\n }\n\n static Optional getLatestCompactionFile(String directory) {\n return Arrays.stream(FileUtils.listDataFiles(new File(directory)))\n .filter(f -> HaloDBFile.findFileType(f) == HaloDBFile.FileType.COMPACTED_FILE)\n .max(Comparator.comparing(File::getName));\n }\n\n static FileTime getFileCreationTime(File file) throws IOException {\n return Files.readAttributes(file.toPath(), BasicFileAttributes.class).creationTime();\n }\n}\n" }, { "path": "src/test/java/com/oath/halodb/TombstoneFileCleanUpTest.java", "content": "package com.oath.halodb;\n\nimport org.testng.Assert;\nimport org.testng.annotations.Test;\n\nimport java.io.File;\nimport java.io.IOException;\nimport java.util.*;\nimport java.util.stream.Collectors;\n\npublic class TombstoneFileCleanUpTest extends TestBase {\n\n @Test(dataProvider = \"Options\")\n public void testDeleteAllRecords(HaloDBOptions options) throws HaloDBException, IOException {\n String directory = TestUtils.getTestDirectory(\"TombstoneFileCleanUpTest\", \"testDeleteAllRecords\");\n\n options.setCleanUpTombstonesDuringOpen(true);\n options.setCompactionThresholdPerFile(1);\n options.setMaxFileSize(1024 * 1024);\n HaloDB db = getTestDB(directory, options);\n\n // 1024 records in 100 files.\n int noOfRecordsPerFile = 1024;\n int noOfFiles = 100;\n int noOfRecords = noOfRecordsPerFile * noOfFiles;\n List records = TestUtils.insertRandomRecordsOfSize(db, noOfRecords, 1024-Record.Header.HEADER_SIZE);\n\n // delete all records\n for (Record r : records) {\n db.delete(r.getKey());\n }\n\n // all files will be deleted except for the last one as it is the current write file. \n TestUtils.waitForCompactionToComplete(db);\n\n // close and open the db.\n db.close();\n db = getTestDBWithoutDeletingFiles(directory, options);\n\n // Since we waited for compaction to complete all but one file must be deleted.\n // Therefore, there will be one tombstone file with 1024 records from the last data file.\n File[] tombstoneFiles = FileUtils.listTombstoneFiles(new File(directory));\n Assert.assertEquals(tombstoneFiles.length, 1);\n TombstoneFile tombstoneFile = new TombstoneFile(tombstoneFiles[0], options, dbDirectory);\n tombstoneFile.open();\n TombstoneFile.TombstoneFileIterator iterator = tombstoneFile.newIterator();\n\n //Make sure that only 1024 tombstones from the last data file are left in the tombstone file after clean up. \n int tombstoneCount = 0;\n List remaining = records.stream().skip(noOfRecords - noOfRecordsPerFile).collect(Collectors.toList());\n while (iterator.hasNext()) {\n TombstoneEntry entry = iterator.next();\n Assert.assertEquals(entry.getKey(), remaining.get(tombstoneCount++).getKey());\n }\n\n Assert.assertEquals(tombstoneCount, noOfRecordsPerFile);\n\n HaloDBStats stats = db.stats();\n Assert.assertEquals(noOfRecords, stats.getNumberOfTombstonesFoundDuringOpen());\n Assert.assertEquals(noOfRecords - noOfRecordsPerFile, stats.getNumberOfTombstonesCleanedUpDuringOpen());\n }\n\n @Test(dataProvider = \"Options\")\n public void testDeleteAndInsertRecords(HaloDBOptions options) throws IOException, HaloDBException {\n String directory = TestUtils.getTestDirectory(\"TombstoneFileCleanUpTest\", \"testDeleteAndInsertRecords\");\n\n options.setCleanUpTombstonesDuringOpen(true);\n options.setCompactionThresholdPerFile(1);\n HaloDB db = getTestDB(directory, new HaloDBOptions());\n\n int noOfRecords = 10_000;\n List records = TestUtils.insertRandomRecords(db, noOfRecords);\n\n // delete all records\n for (Record r : records) {\n db.delete(r.getKey());\n }\n\n // insert records again.\n for (Record r : records) {\n db.put(r.getKey(), r.getValue());\n }\n\n TestUtils.waitForCompactionToComplete(db);\n\n db.close();\n\n // all records were written again after deleting, therefore all tombstone records should be deleted.\n db = getTestDBWithoutDeletingFiles(directory, options);\n\n Assert.assertEquals(FileUtils.listTombstoneFiles(new File(directory)).length, 0);\n\n HaloDBStats stats = db.stats();\n Assert.assertEquals(noOfRecords, stats.getNumberOfTombstonesFoundDuringOpen());\n Assert.assertEquals(noOfRecords, stats.getNumberOfTombstonesCleanedUpDuringOpen());\n }\n\n @Test(dataProvider = \"Options\")\n public void testDeleteRecordsWithoutCompaction(HaloDBOptions options) throws IOException, HaloDBException {\n String directory = TestUtils.getTestDirectory(\"TombstoneFileCleanUpTest\", \"testDeleteRecordsWithoutCompaction\");\n\n options.setMaxFileSize(1024 * 1024);\n options.setCompactionThresholdPerFile(1);\n options.setCleanUpTombstonesDuringOpen(true);\n HaloDB db = getTestDB(directory, options);\n\n // 1024 records in 100 files.\n int noOfRecordsPerFile = 1024;\n int noOfFiles = 100;\n int noOfRecords = noOfRecordsPerFile * noOfFiles;\n List records = TestUtils.insertRandomRecordsOfSize(db, noOfRecords, 1024-Record.Header.HEADER_SIZE);\n\n // delete first record from each file, since compaction threshold is 1 none of the files will be compacted.\n for (int i = 0; i < noOfRecords; i+=noOfRecordsPerFile) {\n db.delete(records.get(i).getKey());\n }\n\n // get the tombstone file. \n File[] originalTombstoneFiles = FileUtils.listTombstoneFiles(new File(directory));\n Assert.assertEquals(originalTombstoneFiles.length, 1);\n\n TestUtils.waitForCompactionToComplete(db);\n\n // close and open db. \n db.close();\n db = getTestDBWithoutDeletingFiles(directory, options);\n\n // make sure that the old tombstone file was deleted. \n Assert.assertFalse(originalTombstoneFiles[0].exists());\n\n // Since none of the files were compacted we cannot delete any of the tombstone records\n // as the stale version of records still exist in the db. \n\n // find the new tombstone file and make sure that all the tombstone records were copied.\n File[] tombstoneFilesAfterOpen = FileUtils.listTombstoneFiles(new File(directory));\n Assert.assertEquals(tombstoneFilesAfterOpen.length, 1);\n Assert.assertNotEquals(tombstoneFilesAfterOpen[0].getName(), originalTombstoneFiles[0].getName());\n\n TombstoneFile tombstoneFile = new TombstoneFile(tombstoneFilesAfterOpen[0], options, dbDirectory);\n tombstoneFile.open();\n TombstoneFile.TombstoneFileIterator iterator = tombstoneFile.newIterator();\n\n int tombstoneCount = 0;\n while (iterator.hasNext()) {\n TombstoneEntry entry = iterator.next();\n Assert.assertEquals(entry.getKey(), records.get(tombstoneCount*1024).getKey());\n tombstoneCount++;\n }\n Assert.assertEquals(tombstoneCount, noOfFiles);\n\n HaloDBStats stats = db.stats();\n Assert.assertEquals(noOfFiles, stats.getNumberOfTombstonesFoundDuringOpen());\n Assert.assertEquals(0, stats.getNumberOfTombstonesCleanedUpDuringOpen());\n }\n\n @Test(dataProvider = \"Options\")\n public void testWithCleanUpTurnedOff(HaloDBOptions options) throws IOException, HaloDBException {\n String directory = TestUtils.getTestDirectory(\"TombstoneFileCleanUpTest\", \"testWithCleanUpTurnedOff\");\n\n options.setCleanUpTombstonesDuringOpen(false);\n options.setCompactionThresholdPerFile(1);\n options.setMaxFileSize(1024 * 1024);\n HaloDB db = getTestDB(directory, new HaloDBOptions());\n\n int noOfRecords = 10_000;\n List records = TestUtils.insertRandomRecords(db, noOfRecords);\n\n // delete all records\n for (Record r : records) {\n db.delete(r.getKey());\n }\n\n // insert records again.\n for (Record r : records) {\n db.put(r.getKey(), r.getValue());\n }\n\n // get the tombstone file.\n File[] originalTombstoneFiles = FileUtils.listTombstoneFiles(new File(directory));\n Assert.assertEquals(originalTombstoneFiles.length, 1);\n\n TestUtils.waitForCompactionToComplete(db);\n\n db.close();\n db = getTestDBWithoutDeletingFiles(directory, options);\n\n // clean up was disabled; tombstone file should be the same. \n File[] tombstoneFilesAfterOpen = FileUtils.listTombstoneFiles(new File(directory));\n Assert.assertEquals(tombstoneFilesAfterOpen.length, 1);\n Assert.assertEquals(tombstoneFilesAfterOpen[0].getName(), originalTombstoneFiles[0].getName());\n\n HaloDBStats stats = db.stats();\n Assert.assertEquals(noOfRecords, stats.getNumberOfTombstonesFoundDuringOpen());\n Assert.assertEquals(0, stats.getNumberOfTombstonesCleanedUpDuringOpen());\n }\n\n @Test(dataProvider = \"Options\")\n public void testCopyMultipleTombstoneFiles(HaloDBOptions options) throws HaloDBException, IOException {\n //Test to make sure that rollover to tombstone files work correctly during cleanup. \n\n String directory = TestUtils.getTestDirectory(\"TombstoneFileCleanUpTest\", \"testCopyMultipleTombstoneFiles\");\n\n options.setCleanUpTombstonesDuringOpen(true);\n options.setCompactionDisabled(true);\n options.setMaxFileSize(512);\n HaloDB db = getTestDB(directory, options);\n\n int noOfRecordsPerFile = 8;\n int noOfFiles = 8;\n int noOfRecords = noOfRecordsPerFile * noOfFiles;\n\n int keyLength = 18;\n int valueLength = 24;\n\n List records = new ArrayList<>();\n for (int i = 0; i < noOfRecords; i++) {\n Record r = new Record(TestUtils.generateRandomByteArray(keyLength), TestUtils.generateRandomByteArray(valueLength));\n records.add(r);\n db.put(r.getKey(), r.getValue());\n }\n\n for (int i = 0; i < noOfRecords/2; i++) {\n Record r = records.get(i);\n db.delete(r.getKey());\n }\n\n // close and open the db.\n db.close();\n db = getTestDBWithoutDeletingFiles(directory, options);\n\n // Since keyLength was 18, plus header length 14, tombstone entry is 32 bytes.\n // Since file size is 512 there would be two tombstone files both of which should be copied.\n File[] tombstoneFiles = FileUtils.listTombstoneFiles(new File(directory));\n Set tombstones = new HashSet<>();\n Assert.assertEquals(tombstoneFiles.length, 2);\n for (File f : tombstoneFiles) {\n TombstoneFile tombstoneFile = new TombstoneFile(f, options, dbDirectory);\n tombstoneFile.open();\n TombstoneFile.TombstoneFileIterator iterator = tombstoneFile.newIterator();\n while (iterator.hasNext()) {\n tombstones.add(Arrays.toString(iterator.next().getKey()));\n }\n }\n\n for (int i = 0; i < tombstones.size(); i++) {\n Assert.assertTrue(tombstones.contains(Arrays.toString(records.get(i).getKey())));\n }\n\n HaloDBStats stats = db.stats();\n Assert.assertEquals(noOfRecords/2, stats.getNumberOfTombstonesFoundDuringOpen());\n Assert.assertEquals(0, stats.getNumberOfTombstonesCleanedUpDuringOpen());\n }\n\n @Test(dataProvider = \"Options\")\n public void testMergeTombstoneFiles(HaloDBOptions options) throws IOException, HaloDBException {\n String directory = TestUtils.getTestDirectory(\"TombstoneFileCleanUpTest\", \"testMergeTombstoneFiles\");\n\n options.setCompactionThresholdPerFile(0.5);\n options.setMaxFileSize(16 * 1024);\n options.setMaxTombstoneFileSize(2 * 1024);\n HaloDB db = getTestDB(directory, options);\n\n // Record size: header 18 + key 18 + value 28 = 64 bytes\n // Tombstone entry size: header 14 + key 18 = 32 bytes\n // Each data file will store 16 * 1024 / 64 = 256 records\n // Each tombstone file will store 2 * 1024 / 32 = 64 entries\n // Total data files 2048 / 256 = 8\n // Total tombstone original file count (1024 / 2 + 1024 / 4) / 64 = 12\n // After cleanup, tombstone file count 1024 / 4 / 64 = 4\n int keyLength = 18;\n int valueLength = 28;\n int noOfRecords = 2048;\n List records = new ArrayList<>();\n for (int i = 0; i < noOfRecords; i++) {\n Record r = new Record(TestUtils.generateRandomByteArray(keyLength), TestUtils.generateRandomByteArray(valueLength));\n records.add(r);\n db.put(r.getKey(), r.getValue());\n }\n\n // The deletion strategy is:\n // Delete total 1/2 records from first half and 1/4 from second half in turn\n // so that each tombstone file contains entries from both parts\n // Because first half has 50% records deleted, the files which hold first\n // half records will be compacted and tombstone entries will be inactive\n // Tombstone entries of second part are still active\n int mid = records.size() / 2;\n for (int i = 0; i < mid; i++) {\n if (i % 2 == 0) {\n db.delete(records.get(i).getKey());\n }\n if (i % 4 == 0) {\n db.delete((records.get(i+mid).getKey()));\n }\n }\n TestUtils.waitForCompactionToComplete(db);\n\n db.close();\n\n File[] original = FileUtils.listTombstoneFiles(new File(directory));\n // See comments above how 12 is calculated\n Assert.assertEquals(original.length, 12);\n\n // disable CleanUpTombstonesDuringOpen, all original tombstone files preserved\n options.setCleanUpTombstonesDuringOpen(false);\n db = getTestDBWithoutDeletingFiles(directory, options);\n\n File[] current = FileUtils.listTombstoneFiles(new File(directory));\n Assert.assertEquals(current.length, original.length);\n for (int i = 0; i < original.length; i++) {\n Assert.assertEquals(current[i].getName(), original[i].getName());\n }\n\n db.close();\n options.setCleanUpTombstonesDuringOpen(true);\n db = getTestDBWithoutDeletingFiles(directory, options);\n\n original = current;\n current = FileUtils.listTombstoneFiles(new File(directory));\n // See comments above how 4 is calculated\n Assert.assertEquals(current.length, 4);\n // All tombstone files are rolled over to new files\n Assert.assertTrue(getFileId(current[0].getName()) > getFileId(original[original.length-1].getName()));\n\n // Delete 1 record and verify currentTombstoneFile is initialized\n db.delete(records.get(1).getKey());\n original = current;\n current = FileUtils.listTombstoneFiles(new File(directory));\n Assert.assertEquals(current.length, original.length + 1);\n db.close();\n }\n\n private int getFileId(String fileName) {\n return Integer.parseInt(fileName.substring(0, fileName.indexOf(\".\")));\n }\n}\n" }, { "path": "src/test/java/com/oath/halodb/TombstoneFileTest.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\npackage com.oath.halodb;\n\nimport org.testng.Assert;\nimport org.testng.annotations.AfterMethod;\nimport org.testng.annotations.BeforeMethod;\nimport org.testng.annotations.Test;\n\nimport java.io.File;\nimport java.io.IOException;\nimport java.nio.ByteBuffer;\nimport java.nio.channels.FileChannel;\nimport java.nio.file.Paths;\nimport java.nio.file.StandardOpenOption;\nimport java.nio.file.attribute.FileTime;\nimport java.util.ArrayList;\nimport java.util.List;\n\npublic class TombstoneFileTest {\n\n private File directory = new File(TestUtils.getTestDirectory(\"TombstoneFileTest\"));\n private DBDirectory dbDirectory;\n private TombstoneFile file;\n private File backingFile;\n private int fileId = 100;\n private FileTime createdTime;\n\n @BeforeMethod\n public void before() throws IOException {\n TestUtils.deleteDirectory(directory);\n dbDirectory = DBDirectory.open(directory);\n file = TombstoneFile.create(dbDirectory, fileId, new HaloDBOptions());\n backingFile = directory.toPath().resolve(file.getName()).toFile();\n createdTime = TestUtils.getFileCreationTime(backingFile);\n try {\n // wait for a second to make sure that the file creation time of the repaired file will be different.\n Thread.sleep(1000);\n } catch (InterruptedException e) {\n }\n }\n\n @AfterMethod\n public void after() throws IOException {\n if (file != null)\n file.close();\n dbDirectory.close();\n TestUtils.deleteDirectory(directory);\n }\n\n @Test\n public void testRepairFileWithCorruptedEntry() throws IOException {\n int noOfRecords = 1000;\n List records = insertTestRecords(noOfRecords);\n\n // add a corrupted entry to the file. \n int sequenceNumber = noOfRecords + 100;\n TombstoneEntry corrupted = new TombstoneEntry(TestUtils.generateRandomByteArray(), sequenceNumber, -1, 21);\n try(FileChannel channel = FileChannel.open(\n Paths.get(directory.getCanonicalPath(), fileId + TombstoneFile.TOMBSTONE_FILE_NAME).toAbsolutePath(), StandardOpenOption.APPEND)) {\n ByteBuffer[] data = corrupted.serialize();\n ByteBuffer header = data[0];\n // change the sequence number, due to which checksum won't match.\n header.putLong(TombstoneEntry.SEQUENCE_NUMBER_OFFSET, sequenceNumber + 100);\n channel.write(data);\n }\n\n TombstoneFile repairedFile = file.repairFile(dbDirectory);\n Assert.assertNotEquals(TestUtils.getFileCreationTime(backingFile), createdTime);\n verifyData(repairedFile, records);\n }\n\n @Test\n public void testRepairFileWithCorruptedKeySize() throws IOException {\n int noOfRecords = 34467;\n List records = insertTestRecords(noOfRecords);\n\n // add a corrupted entry to the file.\n int sequenceNumber = noOfRecords + 100;\n TombstoneEntry corrupted = new TombstoneEntry(TestUtils.generateRandomByteArray(), sequenceNumber, -1, 13);\n try(FileChannel channel = FileChannel.open(\n Paths.get(directory.getCanonicalPath(), fileId + TombstoneFile.TOMBSTONE_FILE_NAME).toAbsolutePath(), StandardOpenOption.APPEND)) {\n ByteBuffer[] data = corrupted.serialize();\n ByteBuffer header = data[0];\n // change the sequence number, due to which checksum won't match.\n header.put(TombstoneEntry.KEY_SIZE_OFFSET, (byte)0xFF);\n channel.write(data);\n }\n\n TombstoneFile repairedFile = file.repairFile(dbDirectory);\n Assert.assertNotEquals(TestUtils.getFileCreationTime(backingFile), createdTime);\n verifyData(repairedFile, records);\n }\n\n @Test\n public void testRepairFileWithIncompleteEntry() throws IOException {\n int noOfRecords = 14;\n List records = insertTestRecords(noOfRecords);\n\n // add a corrupted entry to the file.\n int sequenceNumber = noOfRecords + 100;\n TombstoneEntry corrupted = new TombstoneEntry(TestUtils.generateRandomByteArray(), sequenceNumber, -1, 17);\n try(FileChannel channel = FileChannel.open(\n Paths.get(directory.getCanonicalPath(), fileId + TombstoneFile.TOMBSTONE_FILE_NAME).toAbsolutePath(), StandardOpenOption.APPEND)) {\n ByteBuffer[] data = corrupted.serialize();\n ByteBuffer truncatedKey = ByteBuffer.allocate(corrupted.getKey().length/2);\n truncatedKey.put(TestUtils.generateRandomByteArray(corrupted.getKey().length/2));\n truncatedKey.flip();\n data[1] = truncatedKey;\n channel.write(data);\n }\n\n TombstoneFile repairedFile = file.repairFile(dbDirectory);\n Assert.assertNotEquals(TestUtils.getFileCreationTime(backingFile), createdTime);\n verifyData(repairedFile, records);\n }\n\n private void verifyData(TombstoneFile file, List records) throws IOException {\n TombstoneFile.TombstoneFileIterator iterator = file.newIterator();\n int count = 0;\n while (iterator.hasNext()) {\n TombstoneEntry actual = iterator.next();\n Assert.assertEquals(actual.getKey(), records.get(count).getKey());\n Assert.assertEquals(actual.getSequenceNumber(), records.get(count).getSequenceNumber());\n Assert.assertEquals(actual.getVersion(), records.get(count).getVersion());\n Assert.assertEquals(actual.getCheckSum(), records.get(count).getCheckSum());\n count++;\n }\n\n Assert.assertEquals(count, records.size());\n\n }\n\n private List insertTestRecords(int number) throws IOException {\n List records = new ArrayList<>();\n for (int i = 0; i < number; i++) {\n TombstoneEntry e = new TombstoneEntry(TestUtils.generateRandomByteArray(), i, -1, 1);\n file.write(e);\n records.add(new TombstoneEntry(e.getKey(), e.getSequenceNumber(), e.computeCheckSum(), e.getVersion()));\n }\n return records;\n }\n\n}\n" }, { "path": "src/test/java/com/oath/halodb/UnsTest.java", "content": "/*\n * Copyright 2018, Oath Inc\n * Licensed under the terms of the Apache License 2.0. Please refer to accompanying LICENSE file for terms.\n */\n\n// This code is a derivative work heavily modified from the OHC project. See NOTICE file for copyright and license.\n\npackage com.oath.halodb;\n\nimport org.testng.annotations.AfterMethod;\nimport org.testng.annotations.Test;\nimport sun.misc.Unsafe;\nimport sun.nio.ch.DirectBuffer;\n\nimport java.io.IOException;\nimport java.lang.reflect.Field;\nimport java.nio.ByteBuffer;\nimport java.nio.ByteOrder;\nimport java.util.Random;\n\nimport static org.testng.Assert.*;\n\npublic class UnsTest\n{\n @AfterMethod(alwaysRun = true)\n public void deinit()\n {\n Uns.clearUnsDebugForTest();\n }\n\n private static final Unsafe unsafe;\n\n static final int CAPACITY = 65536;\n static final ByteBuffer directBuffer;\n\n static\n {\n try\n {\n Field field = Unsafe.class.getDeclaredField(\"theUnsafe\");\n field.setAccessible(true);\n unsafe = (Unsafe) field.get(null);\n if (unsafe.addressSize() > 8)\n throw new RuntimeException(\"Address size \" + unsafe.addressSize() + \" not supported yet (max 8 bytes)\");\n\n directBuffer = ByteBuffer.allocateDirect(CAPACITY);\n }\n catch (Exception e)\n {\n throw new AssertionError(e);\n }\n }\n\n private static void fillRandom()\n {\n Random r = new Random();\n directBuffer.clear();\n while (directBuffer.remaining() >= 4)\n directBuffer.putInt(r.nextInt());\n directBuffer.clear();\n }\n\n @Test\n public void testDirectBufferFor() throws Exception\n {\n fillRandom();\n\n ByteBuffer buf = Uns.directBufferFor(((DirectBuffer) directBuffer).address(), 0, directBuffer.capacity(), false);\n\n for (int i = 0; i < CAPACITY; i++)\n {\n byte b = buf.get();\n byte d = directBuffer.get();\n assertEquals(b, d);\n\n assertEquals(buf.position(), directBuffer.position());\n assertEquals(buf.limit(), directBuffer.limit());\n assertEquals(buf.remaining(), directBuffer.remaining());\n assertEquals(buf.capacity(), directBuffer.capacity());\n }\n\n buf.clear();\n directBuffer.clear();\n\n while (buf.remaining() >= 8)\n {\n long b = buf.getLong();\n long d = directBuffer.getLong();\n assertEquals(b, d);\n\n assertEquals(buf.position(), directBuffer.position());\n assertEquals(buf.remaining(), directBuffer.remaining());\n }\n\n while (buf.remaining() >= 4)\n {\n int b = buf.getInt();\n int d = directBuffer.getInt();\n assertEquals(b, d);\n\n assertEquals(buf.position(), directBuffer.position());\n assertEquals(buf.remaining(), directBuffer.remaining());\n }\n\n for (int i = 0; i < CAPACITY; i++)\n {\n byte b = buf.get(i);\n byte d = directBuffer.get(i);\n assertEquals(b, d);\n\n if (i >= CAPACITY - 1)\n continue;\n\n char bufChar = buf.getChar(i);\n char dirChar = directBuffer.getChar(i);\n short bufShort = buf.getShort(i);\n short dirShort = directBuffer.getShort(i);\n\n assertEquals(bufChar, dirChar);\n assertEquals(bufShort, dirShort);\n\n if (i >= CAPACITY - 3)\n continue;\n\n int bufInt = buf.getInt(i);\n int dirInt = directBuffer.getInt(i);\n float bufFloat = buf.getFloat(i);\n float dirFloat = directBuffer.getFloat(i);\n\n assertEquals(bufInt, dirInt);\n assertEquals(bufFloat, dirFloat);\n\n if (i >= CAPACITY - 7)\n continue;\n\n long bufLong = buf.getLong(i);\n long dirLong = directBuffer.getLong(i);\n double bufDouble = buf.getDouble(i);\n double dirDouble = directBuffer.getDouble(i);\n\n assertEquals(bufLong, dirLong);\n assertEquals(bufDouble, dirDouble);\n }\n }\n\n @Test\n public void testAllocate() throws Exception\n {\n long adr = Uns.allocate(100);\n assertNotEquals(adr, 0L);\n Uns.free(adr);\n\n adr = Uns.allocateIOException(100);\n Uns.free(adr);\n }\n\n @Test(expectedExceptions = IOException.class)\n public void testAllocateTooMuch() throws Exception\n {\n Uns.allocateIOException(Long.MAX_VALUE);\n }\n\n @Test\n public void testGetTotalAllocated() throws Exception\n {\n long before = Uns.getTotalAllocated();\n if (before < 0L)\n return;\n\n // TODO Uns.getTotalAllocated() seems not to respect \"small\" areas - need to check that ... eventually.\n// long[] adrs = new long[10000];\n// try\n// {\n// for (int i=0;i before);\n// }\n// finally\n// {\n// for (long adr : adrs)\n// Uns.free(adr);\n// }\n\n long adr = Uns.allocate(128 * 1024 * 1024);\n try\n {\n assertTrue(Uns.getTotalAllocated() > before);\n }\n finally\n {\n Uns.free(adr);\n }\n }\n\n @Test\n public void testCopyMemory() throws Exception\n {\n byte[] ref = HashTableTestUtils.randomBytes(7777 + 130);\n byte[] arr = new byte[7777 + 130];\n\n long adr = Uns.allocate(7777 + 130);\n try\n {\n for (int offset = 0; offset < 10; offset += 13)\n for (int off = 0; off < 10; off += 13)\n {\n Uns.copyMemory(ref, off, adr, offset, 7777);\n\n equals(ref, adr, offset, 7777);\n\n Uns.copyMemory(adr, offset, arr, off, 7777);\n\n equals(ref, arr, off, 7777);\n }\n }\n finally\n {\n Uns.free(adr);\n }\n }\n\n private static void equals(byte[] ref, long adr, int off, int len)\n {\n for (; len-- > 0; off++)\n assertEquals(unsafe.getByte(adr + off), ref[off]);\n }\n\n private static void equals(byte[] ref, byte[] arr, int off, int len)\n {\n for (; len-- > 0; off++)\n assertEquals(arr[off], ref[off]);\n }\n\n @Test\n public void testSetMemory() throws Exception\n {\n long adr = Uns.allocate(7777 + 130);\n try\n {\n for (byte b = 0; b < 13; b++)\n for (int offset = 0; offset < 10; offset += 13)\n {\n Uns.setMemory(adr, offset, 7777, b);\n\n for (int off = 0; off < 7777; off++)\n assertEquals(unsafe.getByte(adr + offset), b);\n }\n }\n finally\n {\n Uns.free(adr);\n }\n }\n\n @Test\n public void testGetLongFromByteArray() throws Exception\n {\n byte[] arr = HashTableTestUtils.randomBytes(32);\n ByteOrder order = directBuffer.order();\n directBuffer.order(ByteOrder.nativeOrder());\n try\n {\n for (int i = 0; i < 14; i++)\n {\n long u = Uns.getLongFromByteArray(arr, i);\n directBuffer.clear();\n directBuffer.put(arr);\n directBuffer.flip();\n long b = directBuffer.getLong(i);\n assertEquals(b, u);\n }\n }\n finally\n {\n directBuffer.order(order);\n }\n }\n\n @Test\n public void testGetPutLong() throws Exception\n {\n long adr = Uns.allocate(128);\n try\n {\n Uns.copyMemory(HashTableTestUtils.randomBytes(128), 0, adr, 0, 128);\n\n for (int i = 0; i < 14; i++)\n {\n long l = Uns.getLong(adr, i);\n assertEquals(unsafe.getLong(adr + i), Uns.getLong(adr, i));\n\n Uns.putLong(adr, i, l);\n assertEquals(unsafe.getLong(adr + i), l);\n }\n }\n finally\n {\n Uns.free(adr);\n }\n }\n\n @Test\n public void testGetPutInt() throws Exception\n {\n long adr = Uns.allocate(128);\n try\n {\n Uns.copyMemory(HashTableTestUtils.randomBytes(128), 0, adr, 0, 128);\n\n for (int i = 0; i < 14; i++)\n {\n int l = Uns.getInt(adr, i);\n assertEquals(unsafe.getInt(adr + i), Uns.getInt(adr, i));\n\n Uns.putInt(adr, i, l);\n assertEquals(unsafe.getInt(adr + i), l);\n }\n }\n finally\n {\n Uns.free(adr);\n }\n }\n\n @Test\n public void testGetPutShort() throws Exception\n {\n long adr = Uns.allocate(128);\n try\n {\n Uns.copyMemory(HashTableTestUtils.randomBytes(128), 0, adr, 0, 128);\n\n for (int i = 0; i < 14; i++)\n {\n short l = Uns.getShort(adr, i);\n assertEquals(unsafe.getShort(adr + i), Uns.getShort(adr, i));\n\n Uns.putShort(adr, i, l);\n assertEquals(unsafe.getShort(adr + i), l);\n }\n }\n finally\n {\n Uns.free(adr);\n }\n }\n\n @Test\n public void testGetPutByte() throws Exception\n {\n long adr = Uns.allocate(128);\n try\n {\n Uns.copyMemory(HashTableTestUtils.randomBytes(128), 0, adr, 0, 128);\n\n for (int i = 0; i < 14; i++)\n {\n ByteBuffer buf = Uns.directBufferFor(adr, i, 8, false);\n byte l = Uns.getByte(adr, i);\n assertEquals(buf.get(0), Uns.getByte(adr, i));\n assertEquals(unsafe.getByte(adr + i), Uns.getByte(adr, i));\n\n Uns.putByte(adr, i, l);\n assertEquals(buf.get(0), l);\n assertEquals(unsafe.getByte(adr + i), l);\n }\n }\n finally\n {\n Uns.free(adr);\n }\n }\n\n @Test\n public void testDecrementIncrement() throws Exception\n {\n long adr = Uns.allocate(128);\n try\n {\n for (int i = 0; i < 120; i++)\n {\n String loop = \"at loop #\" + i;\n long v = Uns.getInt(adr, i);\n Uns.increment(adr, i);\n assertEquals(Uns.getInt(adr, i), v + 1, loop);\n Uns.increment(adr, i);\n assertEquals(Uns.getInt(adr, i), v + 2, loop);\n Uns.increment(adr, i);\n assertEquals(Uns.getInt(adr, i), v + 3, loop);\n Uns.decrement(adr, i);\n assertEquals(Uns.getInt(adr, i), v + 2, loop);\n Uns.decrement(adr, i);\n assertEquals(Uns.getInt(adr, i), v + 1, loop);\n }\n\n Uns.putLong(adr, 8, 1);\n assertTrue(Uns.decrement(adr, 8));\n Uns.putLong(adr, 8, 2);\n assertFalse(Uns.decrement(adr, 8));\n }\n finally\n {\n Uns.free(adr);\n }\n }\n\n @Test\n public void testCompare() throws Exception\n {\n long adr = Uns.allocate(CAPACITY);\n try\n {\n long adr2 = Uns.allocate(CAPACITY);\n try\n {\n\n Uns.setMemory(adr, 5, 11, (byte) 0);\n Uns.setMemory(adr2, 5, 11, (byte) 1);\n\n assertFalse(Uns.memoryCompare(adr, 5, adr2, 5, 11));\n\n assertTrue(Uns.memoryCompare(adr, 5, adr, 5, 11));\n assertTrue(Uns.memoryCompare(adr2, 5, adr2, 5, 11));\n\n Uns.setMemory(adr, 5, 11, (byte) 1);\n\n assertTrue(Uns.memoryCompare(adr, 5, adr2, 5, 11));\n }\n finally\n {\n Uns.free(adr2);\n }\n }\n finally\n {\n Uns.free(adr);\n }\n }\n\n @Test\n public void testCompareManyKeys() {\n\n Random random = new Random();\n for (int i = 0; i < 128; i++) {\n long adr1 = Uns.allocate(CAPACITY);\n long adr2 = Uns.allocate(CAPACITY);\n try {\n byte[] key = com.oath.halodb.TestUtils.generateRandomByteArray();\n Uns.copyMemory(key, 0, adr1, i, key.length);\n Uns.copyMemory(key, 0, adr2, i, key.length);\n assertTrue(Uns.memoryCompare(adr1, i, adr2, i, key.length));\n\n\n int offsetToChange = i + random.nextInt(key.length);\n byte change = (byte)~Uns.getByte(adr2, offsetToChange);\n\n Uns.setMemory(adr2, offsetToChange, 1, change);\n assertFalse(Uns.memoryCompare(adr1, i, adr2, i, key.length));\n }\n finally {\n Uns.free(adr1);\n Uns.free(adr2);\n }\n }\n }\n\n}\n" }, { "path": "src/test/java/com/oath/halodb/histo/EstimatedHistogramTest.java", "content": "package com.oath.halodb.histo;\n\nimport org.testng.Assert;\nimport org.testng.annotations.Test;\nimport org.testng.internal.junit.ArrayAsserts;\n\npublic class EstimatedHistogramTest {\n\n @Test\n public void testGetBuckets() {\n EstimatedHistogram estimatedHistogram = new EstimatedHistogram(\n new long[]{2, 4, 0}, new long[]{1, 2, 6, 0});\n ArrayAsserts.assertArrayEquals(new long[]{1, 2, 6, 0},\n estimatedHistogram.getBuckets(true));\n ArrayAsserts.assertArrayEquals(new long[]{0, 0, 0, 0},\n estimatedHistogram.getBuckets(false));\n }\n\n @Test\n public void testMin() {\n EstimatedHistogram estimatedHistogram = new EstimatedHistogram();\n Assert.assertEquals(estimatedHistogram.min(), 0l);\n\n estimatedHistogram.add(4l);\n estimatedHistogram.add(2l);\n estimatedHistogram.add(3l);\n Assert.assertEquals(estimatedHistogram.min(), 2l);\n }\n\n @Test\n public void testMax() {\n EstimatedHistogram estimatedHistogram1 =\n new EstimatedHistogram(new long[]{2, 4}, new long[]{1, 2, 6});\n\n Assert.assertEquals(estimatedHistogram1.max(), 9223372036854775807L);\n\n EstimatedHistogram estimatedHistogram2 = new EstimatedHistogram();\n Assert.assertEquals(estimatedHistogram2.max(), 0l);\n\n estimatedHistogram2.add(2l);\n estimatedHistogram2.add(4l);\n estimatedHistogram2.add(3l);\n\n Assert.assertEquals(estimatedHistogram2.max(), 4l);\n }\n\n @Test\n public void testPercentile() {\n long[] offsets = new long[]{2, 1, 3, 5, 4};\n long[] bucketData = new long[]{1, 2, 3, 4, 5, 0};\n\n EstimatedHistogram estimatedHistogram =\n new EstimatedHistogram(offsets, bucketData);\n\n Assert.assertEquals(estimatedHistogram.percentile(0.5), 5l);\n Assert.assertEquals(estimatedHistogram.percentile(1.0), 4l);\n Assert.assertEquals(estimatedHistogram.percentile(0.0), 0l);\n }\n\n @Test\n public void testMean() {\n long[] offsets = new long[]{2, 1, 3, 5, 4};\n long[] bucketData = new long[]{1, 2, 3, 4, 5, 0};\n\n EstimatedHistogram estimatedHistogram =\n new EstimatedHistogram(offsets, bucketData);\n Assert.assertEquals(estimatedHistogram.mean(), 4l);\n }\n\n @Test\n public void testIsOverflowed() {\n EstimatedHistogram estimatedHistogram1 =\n new EstimatedHistogram(new long[]{2, 1}, new long[]{1, 2, 0});\n estimatedHistogram1.add(1l);\n\n Assert.assertFalse(estimatedHistogram1.isOverflowed());\n\n EstimatedHistogram estimatedHistogram2 =\n new EstimatedHistogram(new long[]{2}, new long[]{1, 3});\n\n Assert.assertTrue(estimatedHistogram2.isOverflowed());\n }\n\n @Test\n public void testToString() {\n long[] offsets = new long[]{0, 1};\n long[] bucketData = new long[]{1, 2, 1};\n\n EstimatedHistogram estimatedHistogram =\n new EstimatedHistogram(offsets, bucketData);\n\n Assert.assertEquals(estimatedHistogram.toString(),\n \"[-Inf..0]: 1\\n [1..1]: 2\\n [2..Inf]: 1\\n\");\n Assert.assertEquals(new EstimatedHistogram(1).toString(), \"\");\n }\n\n @Test\n public void testEquals() {\n long[] offsets = new long[]{2, 1};\n long[] bucketData = new long[]{1, 2, 0};\n\n EstimatedHistogram estimatedHistogram =\n new EstimatedHistogram(offsets, bucketData);\n\n Assert.assertFalse(estimatedHistogram.equals(\"\"));\n\n Assert.assertTrue(estimatedHistogram.equals(estimatedHistogram));\n Assert.assertTrue(estimatedHistogram.equals(\n new EstimatedHistogram(offsets, bucketData)));\n }\n}\n" }, { "path": "src/test/resources/log4j2-test.xml", "content": "\n\n\n\n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n" } ]