[ { "path": "CODE_OF_CONDUCT.md", "content": "# Code of Conduct\n\nFacebook has adopted a Code of Conduct that we expect project participants to adhere to.\nPlease read the [full text](https://code.fb.com/codeofconduct/)\nso that you can understand what actions will and will not be tolerated.\n" }, { "path": "CONTRIBUTING.md", "content": "# Contributing to this repo\n\n## Pull Requests\n\nIn order to accept your pull request, we need you to submit a CLA. You only need\nto do this once to work on any of Facebook's open source projects.\n\nComplete your CLA here: \n\n## Issues\nWe use GitHub issues to track public bugs. Please ensure your description is\nclear and has sufficient instructions to be able to reproduce the issue.\n\n## License\nBy contributing to this repo, you agree that your contributions will be licensed\nunder the LICENSE file in the root directory of this source tree.\n" }, { "path": "LICENSE", "content": "Attribution-NonCommercial 4.0 International\n\n=======================================================================\n\nCreative Commons Corporation (\"Creative Commons\") is not a law firm and\ndoes not provide legal services or legal advice. Distribution of\nCreative Commons public licenses does not create a lawyer-client or\nother relationship. Creative Commons makes its licenses and related\ninformation available on an \"as-is\" basis. Creative Commons gives no\nwarranties regarding its licenses, any material licensed under their\nterms and conditions, or any related information. Creative Commons\ndisclaims all liability for damages resulting from their use to the\nfullest extent possible.\n\nUsing Creative Commons Public Licenses\n\nCreative Commons public licenses provide a standard set of terms and\nconditions that creators and other rights holders may use to share\noriginal works of authorship and other material subject to copyright\nand certain other rights specified in the public license below. The\nfollowing considerations are for informational purposes only, are not\nexhaustive, and do not form part of our licenses.\n\n Considerations for licensors: Our public licenses are\n intended for use by those authorized to give the public\n permission to use material in ways otherwise restricted by\n copyright and certain other rights. Our licenses are\n irrevocable. Licensors should read and understand the terms\n and conditions of the license they choose before applying it.\n Licensors should also secure all rights necessary before\n applying our licenses so that the public can reuse the\n material as expected. Licensors should clearly mark any\n material not subject to the license. This includes other CC-\n licensed material, or material used under an exception or\n limitation to copyright. More considerations for licensors:\n\twiki.creativecommons.org/Considerations_for_licensors\n\n Considerations for the public: By using one of our public\n licenses, a licensor grants the public permission to use the\n licensed material under specified terms and conditions. If\n the licensor's permission is not necessary for any reason--for\n example, because of any applicable exception or limitation to\n copyright--then that use is not regulated by the license. Our\n licenses grant only permissions under copyright and certain\n other rights that a licensor has authority to grant. Use of\n the licensed material may still be restricted for other\n reasons, including because others have copyright or other\n rights in the material. A licensor may make special requests,\n such as asking that all changes be marked or described.\n Although not required by our licenses, you are encouraged to\n respect those requests where reasonable. More_considerations\n for the public: \n\twiki.creativecommons.org/Considerations_for_licensees\n\n=======================================================================\n\nCreative Commons Attribution-NonCommercial 4.0 International Public\nLicense\n\nBy exercising the Licensed Rights (defined below), You accept and agree\nto be bound by the terms and conditions of this Creative Commons\nAttribution-NonCommercial 4.0 International Public License (\"Public\nLicense\"). To the extent this Public License may be interpreted as a\ncontract, You are granted the Licensed Rights in consideration of Your\nacceptance of these terms and conditions, and the Licensor grants You\nsuch rights in consideration of benefits the Licensor receives from\nmaking the Licensed Material available under these terms and\nconditions.\n\nSection 1 -- Definitions.\n\n a. Adapted Material means material subject to Copyright and Similar\n Rights that is derived from or based upon the Licensed Material\n and in which the Licensed Material is translated, altered,\n arranged, transformed, or otherwise modified in a manner requiring\n permission under the Copyright and Similar Rights held by the\n Licensor. For purposes of this Public License, where the Licensed\n Material is a musical work, performance, or sound recording,\n Adapted Material is always produced where the Licensed Material is\n synched in timed relation with a moving image.\n\n b. Adapter's License means the license You apply to Your Copyright\n and Similar Rights in Your contributions to Adapted Material in\n accordance with the terms and conditions of this Public License.\n\n c. Copyright and Similar Rights means copyright and/or similar rights\n closely related to copyright including, without limitation,\n performance, broadcast, sound recording, and Sui Generis Database\n Rights, without regard to how the rights are labeled or\n categorized. For purposes of this Public License, the rights\n specified in Section 2(b)(1)-(2) are not Copyright and Similar\n Rights.\n d. Effective Technological Measures means those measures that, in the\n absence of proper authority, may not be circumvented under laws\n fulfilling obligations under Article 11 of the WIPO Copyright\n Treaty adopted on December 20, 1996, and/or similar international\n agreements.\n\n e. Exceptions and Limitations means fair use, fair dealing, and/or\n any other exception or limitation to Copyright and Similar Rights\n that applies to Your use of the Licensed Material.\n\n f. Licensed Material means the artistic or literary work, database,\n or other material to which the Licensor applied this Public\n License.\n\n g. Licensed Rights means the rights granted to You subject to the\n terms and conditions of this Public License, which are limited to\n all Copyright and Similar Rights that apply to Your use of the\n Licensed Material and that the Licensor has authority to license.\n\n h. Licensor means the individual(s) or entity(ies) granting rights\n under this Public License.\n\n i. NonCommercial means not primarily intended for or directed towards\n commercial advantage or monetary compensation. For purposes of\n this Public License, the exchange of the Licensed Material for\n other material subject to Copyright and Similar Rights by digital\n file-sharing or similar means is NonCommercial provided there is\n no payment of monetary compensation in connection with the\n exchange.\n\n j. Share means to provide material to the public by any means or\n process that requires permission under the Licensed Rights, such\n as reproduction, public display, public performance, distribution,\n dissemination, communication, or importation, and to make material\n available to the public including in ways that members of the\n public may access the material from a place and at a time\n individually chosen by them.\n\n k. Sui Generis Database Rights means rights other than copyright\n resulting from Directive 96/9/EC of the European Parliament and of\n the Council of 11 March 1996 on the legal protection of databases,\n as amended and/or succeeded, as well as other essentially\n equivalent rights anywhere in the world.\n\n l. You means the individual or entity exercising the Licensed Rights\n under this Public License. Your has a corresponding meaning.\n\nSection 2 -- Scope.\n\n a. License grant.\n\n 1. Subject to the terms and conditions of this Public License,\n the Licensor hereby grants You a worldwide, royalty-free,\n non-sublicensable, non-exclusive, irrevocable license to\n exercise the Licensed Rights in the Licensed Material to:\n\n a. reproduce and Share the Licensed Material, in whole or\n in part, for NonCommercial purposes only; and\n\n b. produce, reproduce, and Share Adapted Material for\n NonCommercial purposes only.\n\n 2. Exceptions and Limitations. For the avoidance of doubt, where\n Exceptions and Limitations apply to Your use, this Public\n License does not apply, and You do not need to comply with\n its terms and conditions.\n\n 3. Term. The term of this Public License is specified in Section\n 6(a).\n\n 4. Media and formats; technical modifications allowed. The\n Licensor authorizes You to exercise the Licensed Rights in\n all media and formats whether now known or hereafter created,\n and to make technical modifications necessary to do so. The\n Licensor waives and/or agrees not to assert any right or\n authority to forbid You from making technical modifications\n necessary to exercise the Licensed Rights, including\n technical modifications necessary to circumvent Effective\n Technological Measures. For purposes of this Public License,\n simply making modifications authorized by this Section 2(a)\n (4) never produces Adapted Material.\n\n 5. Downstream recipients.\n\n a. Offer from the Licensor -- Licensed Material. Every\n recipient of the Licensed Material automatically\n receives an offer from the Licensor to exercise the\n Licensed Rights under the terms and conditions of this\n Public License.\n\n b. No downstream restrictions. You may not offer or impose\n any additional or different terms or conditions on, or\n apply any Effective Technological Measures to, the\n Licensed Material if doing so restricts exercise of the\n Licensed Rights by any recipient of the Licensed\n Material.\n\n 6. No endorsement. Nothing in this Public License constitutes or\n may be construed as permission to assert or imply that You\n are, or that Your use of the Licensed Material is, connected\n with, or sponsored, endorsed, or granted official status by,\n the Licensor or others designated to receive attribution as\n provided in Section 3(a)(1)(A)(i).\n\n b. Other rights.\n\n 1. Moral rights, such as the right of integrity, are not\n licensed under this Public License, nor are publicity,\n privacy, and/or other similar personality rights; however, to\n the extent possible, the Licensor waives and/or agrees not to\n assert any such rights held by the Licensor to the limited\n extent necessary to allow You to exercise the Licensed\n Rights, but not otherwise.\n\n 2. Patent and trademark rights are not licensed under this\n Public License.\n\n 3. To the extent possible, the Licensor waives any right to\n collect royalties from You for the exercise of the Licensed\n Rights, whether directly or through a collecting society\n under any voluntary or waivable statutory or compulsory\n licensing scheme. In all other cases the Licensor expressly\n reserves any right to collect such royalties, including when\n the Licensed Material is used other than for NonCommercial\n purposes.\n\nSection 3 -- License Conditions.\n\nYour exercise of the Licensed Rights is expressly made subject to the\nfollowing conditions.\n\n a. Attribution.\n\n 1. If You Share the Licensed Material (including in modified\n form), You must:\n\n a. retain the following if it is supplied by the Licensor\n with the Licensed Material:\n\n i. identification of the creator(s) of the Licensed\n Material and any others designated to receive\n attribution, in any reasonable manner requested by\n the Licensor (including by pseudonym if\n designated);\n\n ii. a copyright notice;\n\n iii. a notice that refers to this Public License;\n\n iv. a notice that refers to the disclaimer of\n warranties;\n\n v. a URI or hyperlink to the Licensed Material to the\n extent reasonably practicable;\n\n b. indicate if You modified the Licensed Material and\n retain an indication of any previous modifications; and\n\n c. indicate the Licensed Material is licensed under this\n Public License, and include the text of, or the URI or\n hyperlink to, this Public License.\n\n 2. You may satisfy the conditions in Section 3(a)(1) in any\n reasonable manner based on the medium, means, and context in\n which You Share the Licensed Material. For example, it may be\n reasonable to satisfy the conditions by providing a URI or\n hyperlink to a resource that includes the required\n information.\n\n 3. If requested by the Licensor, You must remove any of the\n information required by Section 3(a)(1)(A) to the extent\n reasonably practicable.\n\n 4. If You Share Adapted Material You produce, the Adapter's\n License You apply must not prevent recipients of the Adapted\n Material from complying with this Public License.\n\nSection 4 -- Sui Generis Database Rights.\n\nWhere the Licensed Rights include Sui Generis Database Rights that\napply to Your use of the Licensed Material:\n\n a. for the avoidance of doubt, Section 2(a)(1) grants You the right\n to extract, reuse, reproduce, and Share all or a substantial\n portion of the contents of the database for NonCommercial purposes\n only;\n\n b. if You include all or a substantial portion of the database\n contents in a database in which You have Sui Generis Database\n Rights, then the database in which You have Sui Generis Database\n Rights (but not its individual contents) is Adapted Material; and\n\n c. You must comply with the conditions in Section 3(a) if You Share\n all or a substantial portion of the contents of the database.\n\nFor the avoidance of doubt, this Section 4 supplements and does not\nreplace Your obligations under this Public License where the Licensed\nRights include other Copyright and Similar Rights.\n\nSection 5 -- Disclaimer of Warranties and Limitation of Liability.\n\n a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE\n EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS\n AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF\n ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS,\n IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION,\n WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR\n PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS,\n ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT\n KNOWN OR DISCOVERABLE. WHERE DISCLAIMERS OF WARRANTIES ARE NOT\n ALLOWED IN FULL OR IN PART, THIS DISCLAIMER MAY NOT APPLY TO YOU.\n\n b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE\n TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION,\n NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT,\n INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES,\n COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR\n USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN\n ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR\n DAMAGES. WHERE A LIMITATION OF LIABILITY IS NOT ALLOWED IN FULL OR\n IN PART, THIS LIMITATION MAY NOT APPLY TO YOU.\n\n c. The disclaimer of warranties and limitation of liability provided\n above shall be interpreted in a manner that, to the extent\n possible, most closely approximates an absolute disclaimer and\n waiver of all liability.\n\nSection 6 -- Term and Termination.\n\n a. This Public License applies for the term of the Copyright and\n Similar Rights licensed here. However, if You fail to comply with\n this Public License, then Your rights under this Public License\n terminate automatically.\n\n b. Where Your right to use the Licensed Material has terminated under\n Section 6(a), it reinstates:\n\n 1. automatically as of the date the violation is cured, provided\n it is cured within 30 days of Your discovery of the\n violation; or\n\n 2. upon express reinstatement by the Licensor.\n\n For the avoidance of doubt, this Section 6(b) does not affect any\n right the Licensor may have to seek remedies for Your violations\n of this Public License.\n\n c. For the avoidance of doubt, the Licensor may also offer the\n Licensed Material under separate terms or conditions or stop\n distributing the Licensed Material at any time; however, doing so\n will not terminate this Public License.\n\n d. Sections 1, 5, 6, 7, and 8 survive termination of this Public\n License.\n\nSection 7 -- Other Terms and Conditions.\n\n a. The Licensor shall not be bound by any additional or different\n terms or conditions communicated by You unless expressly agreed.\n\n b. Any arrangements, understandings, or agreements regarding the\n Licensed Material not stated herein are separate from and\n independent of the terms and conditions of this Public License.\n\nSection 8 -- Interpretation.\n\n a. For the avoidance of doubt, this Public License does not, and\n shall not be interpreted to, reduce, limit, restrict, or impose\n conditions on any use of the Licensed Material that could lawfully\n be made without permission under this Public License.\n\n b. To the extent possible, if any provision of this Public License is\n deemed unenforceable, it shall be automatically reformed to the\n minimum extent necessary to make it enforceable. If the provision\n cannot be reformed, it shall be severed from this Public License\n without affecting the enforceability of the remaining terms and\n conditions.\n\n c. No term or condition of this Public License will be waived and no\n failure to comply consented to unless expressly agreed to by the\n Licensor.\n\n d. Nothing in this Public License constitutes or may be interpreted\n as a limitation upon, or waiver of, any privileges and immunities\n that apply to the Licensor or You, including from the legal\n processes of any jurisdiction or authority.\n\n=======================================================================\n\nCreative Commons is not a party to its public\nlicenses. Notwithstanding, Creative Commons may elect to apply one of\nits public licenses to material it publishes and in those instances\nwill be considered the “Licensor.” The text of the Creative Commons\npublic licenses is dedicated to the public domain under the CC0 Public\nDomain Dedication. Except for the limited purpose of indicating that\nmaterial is shared under a Creative Commons public license or as\notherwise permitted by the Creative Commons policies published at\ncreativecommons.org/policies, Creative Commons does not authorize the\nuse of the trademark \"Creative Commons\" or any other trademark or logo\nof Creative Commons without its prior written consent including,\nwithout limitation, in connection with any unauthorized modifications\nto any of its public licenses or any other arrangements,\nunderstandings, or agreements concerning use of licensed material. For\nthe avoidance of doubt, this paragraph does not form part of the\npublic licenses.\n\nCreative Commons may be contacted at creativecommons.org.\n" }, { "path": "README.md", "content": "# PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them\n\n\nThis repository contains code and models to support the research paper [PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them](https://arxiv.org/abs/2102.07033)\n\n
\n

\n \"Facebook\n
\n

\n
\n\n## Table of Contents\n\n* [Table of Contents](#table-of-contents)\n* [Data Downloads](#data-downloads)\n * [PAQ QA-pairs](#paq-qa-pairs)\n * [PAQ Metadata](#paq-metadata)\n * [Preprocessed Wikipedia Dump](#preprocessed-wikipedia-dump)\n * [Passage Selector Scores](#passage-selector-scores)\n * [PAQ QA-pair metadata](#paq-qa-pair-metadata)\n * [PAQ unfiltered QA-pair metadata](#paq-unfiltered-qa-pair-metadata)\n * [Training/Dev/Test QA Pairs](#trainingdevtest-qa-pairs)\n* [Code and Models](#code-and-models)\n * [Installation and Setup:](#installation-and-setup)\n * [Download Tool](#download-tool)\n * [Question Answering with RePAQ](#question-answering-with-repaq)\n * [RePAQ Retrievers:](#repaq-retrievers)\n * [Minimal Retrieval Inference Example:](#minimal-retrieval-inference-example)\n * [Retriever Models, Precomputed Vectors and Indexes:](#retriever-models-precomputed-vectors-and-indexes)\n * [Embedding QA pairs:](#embedding-qa-pairs)\n * [Building indices:](#building-indices)\n * [Retriever Inference:](#retriever-inference)\n * [Evaluating Retriever Results:](#evaluating-retriever-results)\n * [RePAQ ReRankers:](#repaq-rerankers)\n * [Minimal Reranker Inference example:](#minimal-reranker-inference-example)\n * [Reranker Models:](#reranker-models)\n * [ReRanker Inference:](#reranker-inference)\n * [Evaluating Rerankers:](#evaluating-rerankers)\n * [Question-Answer Pair Generation](#question-answer-pair-generation)\n * [Passage Scoring/Ranking](#passage-scoringranking)\n * [Answer Extraction](#answer-extraction)\n * [Question Generation](#question-generation)\n * [Filtering Generated QA-pairs](#filtering-generated-qa-pairs)\n * [End2End Generation Tool](#end2end-generation-tool)\n* [Citing](#citing)\n* [LICENSE](#license)\n * [Code License:](#code-license)\n * [Data License:](#data-license)\n \n## Data Downloads\n\nPAQ QA pairs, their metadata, preprocessed wikipedia dumps and Train/dev/test QA pairs downloads are described in this section. For downloading models, indices etc, see [Code And Models](#code-and-models) section.\nIn addition to downloading the data here, you can use the `paq.download` tool, (recommended for downloading models, indices etc), see the [Download Tool](#download-tool) section for use.\n\n### PAQ QA-pairs\n\nThe PAQ QA pairs can be downloaded below. We use the same format as for NQ-open (see [here](https://github.com/google-research-datasets/natural-questions/tree/master/nq_open)). The\n TQA_TRAIN_NQ_TRAIN_PAQ is the concatenation of the TriviaQA and NQ training QA-Pairs with the PAQ QA-Pairs.\n\n| Dataset | # QAs | Size (unzipped)| link | License |\n| ------------- | ------------- | --------- | ---- | -----|\n| PAQ | 64.9M | 5.8 GB| [download](https://dl.fbaipublicfiles.com/paq/v1/PAQ.tar.gz) | [CC-BY-SA](https://creativecommons.org/licenses/by-sa/3.0/)|\n| PAQ-L1 | 14.1M | 1.3 GB| [download](https://dl.fbaipublicfiles.com/paq/v1/PAQ_L1.tar.gz) | [CC-BY-SA](https://creativecommons.org/licenses/by-sa/3.0/)|\n| PAQ-L4 | 53.8M | 4.9 GB| [download](https://dl.fbaipublicfiles.com/paq/v1/PAQ_L4.tar.gz) | [CC-BY-SA](https://creativecommons.org/licenses/by-sa/3.0/)|\n| PAQ-NE1 | 12.0M | 1.0 GB| [download](https://dl.fbaipublicfiles.com/paq/v1/PAQ_NE1.tar.gz) | [CC-BY-SA](https://creativecommons.org/licenses/by-sa/3.0/)|\n| TQA_TRAIN_NQ_TRAIN_PAQ | 65.00M | 5.9 GB| [download](https://dl.fbaipublicfiles.com/paq/v1/TQA_TRAIN_NQ_TRAIN_PAQ.tar.gz) | [CC-BY-SA](https://creativecommons.org/licenses/by-sa/3.0/)|\n\n\n### PAQ Metadata\n\nAvailable metadata to support PAQ is available, and can be downloaded from the following table. See the descriptions below for details:\n\n| Dataset | Size (unzipped)| link | License |\n| ------------- | ------------- | --------- | ----|\n| Preprocessed Wikipedia Dump | 13 GB| [download](https://dl.fbaipublicfiles.com/paq/v1/psgs_w100.tsv.gz) | [CC-BY-SA](https://creativecommons.org/licenses/by-sa/3.0/)|\n| Passage Selector Scores | 560 MB| [download](https://dl.fbaipublicfiles.com/paq/v1/PASSAGE_SCORES.tar.gz) | [CC-BY-SA](https://creativecommons.org/licenses/by-sa/3.0/)|\n| PAQ QA-Pair metadata | 16 GB| [download](https://dl.fbaipublicfiles.com/paq/v1/PAQ.metadata.jsonl.gz) | [CC-BY-SA](https://creativecommons.org/licenses/by-sa/3.0/)|\n| PAQ *unfiltered* QA-pairs and metadata | 95 GB| [download](https://dl.fbaipublicfiles.com/paq/v1/PAQ.unfiltered_metadata.jsonl.gz) | [CC-BY-SA](https://creativecommons.org/licenses/by-sa/3.0/)|\n\n### Preprocessed Wikipedia Dump\n\nThis file contains the preprocessed wikipedia dump used to generate PAQ. The file consists of 100-word passages of a 2018 Wikipedia dump, and was produced by [Karphukin et al.](https://github.com/facebookresearch/DPR) for [DPR](https://github.com/facebookresearch/DPR).\nThe file is in TSV format, with 3 columns. The first column is passage id, the second column is the passage text, the third is the wikipedia article title. \n\n### Passage Selector Scores\n\nThis file contains the passage selection scores for passages, using the passage selection model described in the paper.\nThe file is in TSV format, with 2 columns. The first column is passage id (see \"Preprocessed Wikipedia Dump\"), the second column is the logprob score from the passage selector for that passage.\n\n### PAQ QA-pair metadata\n\nThis file contains metadata for the QA pairs in PAQ. The file is in jsonl format. Each line is a json dict with metadata for one question-answer pair in PAQ.\nThe format is as follows:\n```\n{\n \"question\": question string\n \"subsets\": list of PAQ subsets the question appears in (\"L1\", \"L4\" or \"NE\")\n \"answer\": the question's answer produced by the consistency filter model\n \"passage_score\": passage selection score of highest scoring passage that generated this question\n \"answers\": [\n {\n \"passage_id\": id of wiki passage this answer was extracted from (see \"Preprocessed Wikipedia Dump\")\n \"offset\": character offset to start of answer span\n \"text\": text of answer span\n \"extractor\": answer extractor model, either \"L\" (for learnt extracor), or \"NE\" (for Named Entity extractor)\n },\n ...\n ]\n}\n```\nThere are a small number of questions where the \"subset\" is \"NE-legacy\". These questions were generated by an earlier iteration of the \"NE\" generation pipeline.\n\n### PAQ *unfiltered* QA-pair metadata\n\nThis file contains similar metadata to that described above in \"PAQ QA pair metadata\", but for *all* generated questions, even those that do not pass the consistency filter. \nAs such, this is a very large file, and is provided for completeness, but should not be of interest to most users interested in PAQ metadata.\nThe file is in jsonl format. Each line is a json dict with metadata for one question-answer pair.\nThe format is as follows:\n```\n{\n \"question\": question string\n \"subsets\": list of PAQ subsets the question appears in (\"L1\", \"L4\" or \"NE\")\n \"consistent_subsets\": list of PAQ subsets the question appears in, which pass the consistnency filters (\"L1\", \"L4\" or \"NE\")\n \"canonical_answer\": the question's answer produced by the consistency filter model\n \"consistent\": boolean. If true, the question passes the global consistency filter\n \"passage_score\": passage selection score of highest scoring passage that generated this question\n \"answers\": [\n {\n \"passage_id\": id of wiki passage this answer was extracted from (see \"Preprocessed Wikipedia Dump\")\n \"offset\": character offset to start of answer span\n \"text\": text of answer span\n \"extractor\": answer extractor model, either \"L\" (for learnt extracor), or \"NE\" (for Named Entity extractor)\n \"consistent\": boolean. If true, this answer span is the consistent with the answer from the global consistency filter\n },\n ...\n ]\n}\n```\n\n### Training/Dev/Test QA Pairs\n\nThe QA Pairs in the Open Domain NaturalQuestions and TriviaQA Train/Dev/Test sets are available below, as well as a file with the concatenation of the training sets and PAQ (useful for retrieval later).\n\n\n| Dataset | Description | Link | \n| ------------- |------------- | --------- |\n| NQ-open.train-train.jsonl | Open-NaturalQuestions Training set | [download](https://dl.fbaipublicfiles.com/paq/v1/annotated_datasets/NQ-open.train-train.jsonl)|\n| NQ-open.train-dev.jsonl | Open-NaturalQuestions Development set| [download](https://dl.fbaipublicfiles.com/paq/v1/annotated_datasets/NQ-open.train-dev.jsonl)|\n| NQ-open.test.jsonl | Open-NaturalQuestions Test set| [download](https://dl.fbaipublicfiles.com/paq/v1/annotated_datasets/NQ-open.test.jsonl)|\n| triviaqa.train-train.jsonl | Open-TriviaQA Training set | [download](https://dl.fbaipublicfiles.com/paq/v1/annotated_datasets/triviaqa.train-train.jsonl)|\n| triviaqa.train-dev.jsonl | Open-TriviaQA Development set| [download](https://dl.fbaipublicfiles.com/paq/v1/annotated_datasets/triviaqa.train-dev.jsonl)|\n| triviaqa.test.jsonl | Open-TriviaQA Test set| [download](https://dl.fbaipublicfiles.com/paq/v1/annotated_datasets/triviaqa.test.jsonl)|\n| tqa-train-nq-train-PAQ.jsonl | Concatenation of NQ-open.train-train.jsonl, triviaqa.train-train.jsonl and PAQ | [download](https://dl.fbaipublicfiles.com/paq/v1/TQA_TRAIN_NQ_TRAIN_PAQ.tar.gz)|\n\n## Code and Models\n\nAll users should follow the instructions in [Installation and Setup](#installation-and-setup), and use the [Download Tool](#download-tool), which will make downloanding models and assets much easier.\n\nCode to run inference for Question Answering using RePAQ and the full question generation pipeline are now available. Functionality to help train your own models is coming soon.\n\nUsers interested in running question answering with REPAQ, read [Question Answering with RePAQ](#question-answering-with-repaq).\n\nUsers interested in running Question generation using the PAQ generation pipeline, read [Question Answering with RePAQ](#question-answering-with-repaq).\n\n\n\n### Installation and Setup:\n\nWe highly recommend you use conda environments. The requirements are pytorch, spacy, Transformers 4.1.0 (other versions unlikely to work), FID, and the packages listed in `requirements.txt`.\nThe following script should install all nececessary code dependencies:\n\n```bash\nconda create -n paq python=3.7\nconda activate paq\n\n# install pytorch\nconda install pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 cudatoolkit=10.1 -c pytorch\nconda install -c pytorch faiss-gpu cudatoolkit=10.1\n\n# For Spacy:\nconda install -c conda-forge spacy\nconda install -c conda-forge cupy\npython -m spacy download en_core_web_sm\npip install -r requirements.txt\n\n# Install FID for QA-pair consistency filtering:\ngit clone git@github.com:facebookresearch/FiD.git\ncd FiD; git checkout baf533c3f7a26c1cac624ee9252ce5ccf344a935\n\n```\n\n### Download Tool\n\nTo make downloading resources easier, we've built a download tool.\nThis is the recommended way for downloading data, trained models, precomputed vectors and indices. \nThis will download and uncompress resources to the `./data` directory, where the code will expect these resources to be, and handle path management.\nRun it by supplying a resource key name (run with `-h` to see available resources): \n```bash\n# Downloads a RePAQ retriever model:\n$ python -m paq.download -v -n models.retrievers.retriever_multi_base_256\n```\n\n### Question Answering with RePAQ\n\nQuestion Answering over PAQ with RePAQ is accomplished using [Dense Retrieval](#repaq-retrievers), optionally following by [Reranking](#repaq-rerankers). \nReranking will improve accuracy, but is slower.\n\nTo enable wider use our work,\nwe have trained more compact retrievers and indices than those used in the original paper.\nThse will still give strong results, but run on machines with smaller GPUs and modest amounts of CPU RAM (64GB CPU RAM should be plenty). \nThese models are only marginally less accurate than the larger ones used in the paper, and we list them as \"recommended\" in the tables below.\n\n\n\n#### RePAQ Retrievers:\n\n##### Minimal Retrieval Inference Example:\nTL;DR if you just want to run retrieval:\n\nFirst, download 1) A retrieval model, 2) A KB of QA Pairs (in our case, TQA train set, NQ train set and PAQ) and 3) a pre-built index for those QA Pairs.\n\n```bash\n# download retriever model\n$ python -m paq.download -v -n models.retrievers.retriever_multi_base_256\n\n# Download QA Pairs, and a corresponding faiss index:\n$ python -m paq.download -v -n paq.TQA_TRAIN_NQ_TRAIN_PAQ\n$ python -m paq.download -v -n indices.multi_base_256_hnsw_sq8\n\n# Download NaturalQuestions data, we'll run inference on the test set\n$ python -m paq.download -v -n annotated_datasets.naturalquestions\n\n```\n\nThen, run retrieval inference (here we're using the v fast but slightly less accurate HNSW faiss index):\n\n```bash\n$ python -m paq.retrievers.retrieve \\\n --model_name_or_path ./data/models/retrievers/retriever_multi_base_256 \\\n --qas_to_answer data/annotated_datasets/NQ-open.test.jsonl \\\n --qas_to_retrieve_from ./data/paq/TQA_TRAIN_NQ_TRAIN_PAQ/tqa-train-nq-train-PAQ.jsonl \\\n --top_k 50 \\\n --output_file my_retrieval_results.jsonl \\\n --faiss_index_path data/indices/multi_base_256_hnsw_sq8.faiss \\\n --fp16 \\\n --memory_friendly_parsing \\\n --verbose\n```\n\n\nFinally, either use a reranker to rerank the top K results (see [here](#minimal-reranker-inference-example)), or evaluate retrieval performance:\n\n```bash\n$ python -m paq.evaluation.eval_retriever \\\n --predictions my_retrieval_results.jsonl \\\n --references data/annotated_datasets/NQ-open.test.jsonl \\\n --hits_at_k 1,10,50\n1: 40.0%\n(1443 / 3610)\n10: 55.7%\n(2010 / 3610)\n50: 63.9%\n(2306 / 3610)\n```\n\n\n##### Retriever Models, Precomputed Vectors and Indexes:\n\nThe following table lists the recommended models for inference. \nFor an exahustive list of models available, see [full_models_list.md](./full_models_list.md). \nWe highly recommend using `retriever_multi_base_256`. \nThis model has been designed to be compute and memory-friendly. \nIt's embedding dimension is 256 c.f. 768 used in the original paper, saving RAM when performing retrieval.\nIt outperforms the base model from the paper, and loses only 0.7% on average ove the xlarge model from the paper.\n\n\n| Model | Training data | Architecture | Embedding Dim | NQ EM | + rerank | TQA EM | + rerank | Download Resource Key Name |\n| ------------- |----------| --- | --------- | ---------- |---- |---- | ---- | ---- |\n| retriever_multi_base_256 (recommended)| NQ + TQA | AlBERT-base | 256 | 41.4 | 47.3 | 40.2 | 50.9| `models.retrievers.retriever_multi_base_256` |\n| retriever_multi_base | NQ + TQA | AlBERT-base | 728 | 40.9| 47.4 | 39.7 | 51.2 | `models.retrievers.retriever_multi_base` |\n| retriever_multi_xlarge | NQ + TQA | AlBERT-xlarge| 728 | 41.7 | 47.6 | 41.3 | 52.1 |`models.retrievers.retriever_multi_xlarge`|\n\nThe table below lists available precomputed embeddings and indices for download. The embeddings are stored according to the order in tqa-train-nq-train-PAQ.jsonl, corresponding to the TQA training set, the NQ training set and PAQ.\nI.e. the kth QA pair in the file is embedded in the kth vector in these files.\n\nTo download precomputed vectors, use the `paq/download.py` script, as indicated in the table. \nWe recommend using the FAISS indexes for running inference, either `multi_base_256.flat.sq8.faiss`\n(slower, 1-10s questions/sec, but more accurate, and has lowest memory requirement ~16GB RAM), \nor `multi_base_256.hnsw.sq8.faiss` (very fast, 100-1000s questions/sec depending on machine, slightly less accurate (0.8% on average) but higher memory requirements ~32GB RAM)\n\n\n| File | Description | Size | Download Resource Key Name | \n| ------------- |------------- | --------- |---- |\n| tqa-train-nq-train-PAQ.jsonl (required) | Concatenation of NQ-open.train-train.jsonl, triviaqa.train-train.jsonl and PAQ | | `paq.TQA_TRAIN_NQ_TRAIN_PAQ` |\n| multi_base_256_vectors | embeddings for QAS in `tqa-train-nq-train-PAQ.jsonl` using `retriever_multi_base_256` | 16GB | `vectors.multi_base_vectors_256 ` |\n| multi_base_vectors | embeddings for QAS in `tqa-train-nq-train-PAQ.jsonl` using `retriever_multi_base` | 48GB |`vectors.multi_base_vectors` |\n| multi_xlarge_vectors| embeddings for QAS in `tqa-train-nq-train-PAQ.jsonl` using `retriever_multi_xlarge` | 48GB| `vectors.multi_xlarge_vectors` |\n| multi_base_256.flat.sq8.faiss (recommended) | Flat FAISS index for `retriever_multi_base_256` - slower (1-10s questions / sec) | 16GB | `indices.multi_base_256_flat_sq8.faiss`|\n| multi_base_256.hnsw.sq8.faiss (recommended) | Fast FAISS index for `retriever_multi_base_256` - faster (100-1000s queries / sec) | 32GB | `indices.multi_base_256_hnsw_sq8.faiss`|\n\n\n\n##### Embedding QA pairs:\n\nTo embed a set of QA pairs in the NaturalQuestions jsonl format, use the `paq/evaluation/embed.py` file.\n\nE.g. to embed the NQ training set to vectors using the `retriever_multi_base_256` model, and write them to disk,\nrun the following command:\n\n```\npython -m paq.retrievers.embed \\\n --model_name_or_path ./data/models/retrievers/retriever_multi_base_256 \\\n --qas_to_embed data/annotated_datasets/NQ-open.train-train.jsonl \\\n --output_dir ./my_vectors \\\n --fp16 \\\n --batch_size 128 \\\n --verbose \\\n --n_jobs -1 \n# see below for explanation of --n_jobs\n```\n\nFor very large numbers of QA pairs, you may want to run this in parallel. \nThis script is set up to work with submitit, and by default, can submit a slurm job array to embed the QA pairs in parallel.\nFor example, to run embedding locally, set `--n_jobs -1` (As above), or to run 10 parallel jobs to embed a file, run with `--n_jobs 10`.\nThe full command is given below:\n\n```\npython -m paq.retrievers.embed \\\n --model_name_or_path ./data/models/retrievers/retriever_multi_base_256 \\\n --qas_to_embed data/annotated_datasets/NQ-open.train-train.jsonl \\\n --output_dir ./my_vectors_distributed \\\n --fp16 \\\n --batch_size 128 \\\n --verbose \\\n --memory_friendly_parsing \\\n --n_jobs 10 \\\n --slurm_partition my_clusters_partition \\\n --slurm_comment \"my embedding job\"\n \n```\n\nThe submitit job array config can be seen and edited for your clusters needs at `paq/paq_utils.py` (the `get_submitit_executor` function)\n\n##### Building indices:\n\nTo build faiss MIPS indices on vectors produced by `paq.retrievers.embed`, (for improved quantization and speed over raw exact search in pytorch), use the `paq/retreiver/build_index.py`.\nThis will allow you to build indices like the ones used in the paper (specifically, Flat and HNSW indices, optionally with scalar quantization).\n\n```\n# build a flat index with Scaler quantization (slower queries, but slightly more accurate)\npython -m paq.retrievers.build_index \\\n --embeddings_dir ./my_vectors \\\n --output_path ./my_index.faiss \\\n --SQ8 \\\n --verbose\n\n# or, build an hnsw index with scaler (mcuh much faster qurerying, slightly less accurate)\npython -m paq.retrievers.build_index \\\n --embeddings_dir ./my_vectors \\\n --output_path ./my_index.hnsw.faiss \\\n --hnsw \\\n --SQ8 \\\n --store_n 32 \\\n --ef_construction 128 \\\n --ef_search 128 \\\n --verbose\n\n```\n\nBuilding indices is a deep, nuanced and complex area. The scripts we provide to build indices is mostly a convenience and reproduciblity wrapper.\nIt's likely that stronger compression is possible without losing performance (e.g. by using Product Quantization), as is faster inference. \nIf the indexes we provide are too large or slow, consider building your own by referring the the [faiss documentation](https://github.com/facebookresearch/faiss) directly.\n\n##### Retriever Inference:\n\nRun QA-pair Retrieval using `paq/retrievers/retrieve.py`. You can see argument help by passing `-h`. \nYou must pass in a jsonl file of QA pairs to retrieve from, using the `--qas_to_retrieve_from` argument. \nYou can also pass in either a directory of embeddings for the qa-pairs to retrieve from using the `--precomputed_embeddings_dir` (e.g. the output of `paq.retrievers.embed`) \nor a faiss index of the qa-pairs to retrieve from, using the `--faiss_index_path`. If neither `--faiss_index_path` or `--precomputed_embeddings_dir` are given, the QA-pairs to retrieve from will be embedded on-the-fly. This may be slow for large QA-pair KBs.\n\nThe following command will run retrieve the top 50 QA-pairs from the PAQ KB for the NQ-test set, using the fast HNSW faiss index, and write the results to `my_retrieval_results.jsonl`\n\n```bash\n#Download the relevant artefacts\n$ python -m paq.download -v -n models.retrievers.retriever_multi_base_256\n$ python -m paq.download -v -n paq.TQA_TRAIN_NQ_TRAIN_PAQ\n$ python -m paq.download -v -n indices.multi_base_256_hnsw_sq8\n$ python -m paq.download -v -n annotated_datasets.naturalquestions\n\n$ python -m paq.retrievers.retrieve \\\n --model_name_or_path ./data/models/retrievers/retriever_multi_base_256 \\\n --qas_to_answer data/annotated_datasets/NQ-open.test.jsonl \\\n --qas_to_retrieve_from ./data/paq/TQA_TRAIN_NQ_TRAIN_PAQ/tqa-train-nq-train-PAQ.jsonl \\\n --top_k 50 \\\n --output_file my_retrieval_results.jsonl \\\n --faiss_index_path data/indices/multi_base_256_hnsw_sq8.faiss \\\n --fp16 \\\n --memory_friendly_parsing \\\n --verbose\n```\n \n##### Evaluating Retriever Results:\n\nEvaluate retrieval performance using the `paq.evaluation.eval_retriever` tool. \nIt will return the hits@k (whether the correct answer is in the top K retrieved questions' answers). Hits@1 is equivalent to Exact Match score\n\n```bash\n$ python -m paq.evaluation.eval_retriever \\\n --predictions my_retrieval_results.jsonl \\\n --references data/annotated_datasets/NQ-open.test.jsonl \\\n --hits_at_k 1,10,50\n1: 40.0%\n(1443 / 3610)\n10: 55.7%\n(2010 / 3610)\n50: 63.9%\n(2306 / 3610)\n```\n\n\n#### RePAQ ReRankers:\n\n##### Minimal Reranker Inference example:\nTl;DR for if you just want to run reranking:\nFirst, download a reranker model, (and if you dont already have retrieval results you want to rerank, download some)\n\n```bash\n# download reranker model (here we're using the albert xxlarge model, smaller ones are available)\n$ python -m paq.download -v -n models.rerankers.reranker_multi_xxlarge\n\n# download some retrieval results to rerank if you dont already have some\n$ python -m paq.download -v -n predictions.retriever_results.multi_xlarge_nq\n\n```\nNext, run reranking:\n```\n$ python -m paq.rerankers.rerank \\\n --model_name_or_path data/models/rerankers/reranker_multi_xxlarge \\\n --qas_to_rerank data/predictions/retriever_results/multi_xlarge_nq_test.jsonl \\\n --output_file my_reranker_results.jsonl \\\n --top_k 50 \\\n --fp16 \\\n --batch_size 4 --verbose --n_jobs -1\n```\n\nThen calculate results:\n```\n$ python -m paq.evaluation.eval_reranker --predictions my_reranker_results.jsonl --references data/annotated_datasets/NQ-open.test.jsonl\n47.6%\n(1699 / 3610)\n```\n\n\n##### Reranker Models:\n\nThe following table lists the recommended models for inference. \nFor an exahustive list of models available, see [full_models_list.md](./full_models_list.md). \n\n| Model | Training data | Architecture | NQ EM | TQA EM | Download Resource Key Name |\n| ------------- |----------| --- | --------- | ---------- |---- |\n|reranker_multi_base| NQ + TQA| AlBERT-base |46.0 |48.9 | `models.rerankers.reranker_multi_base`| \n|reranker_multi_large| NQ + TQA|AlBERT-large | 46.2| 49.4|`models.rerankers.reranker_multi_large`| \n|reranker_multi_xlarge| NQ + TQA|AlBERT-xlarge | 46.0| 49.1| `models.rerankers.reranker_multi_xlarge`| \n|reranker_multi_xxlarge| NQ + TQA|AlBERT-xxlarge | 47.7| 52.1 | `models.rerankers.reranker_multi_xxlarge`| \n\n##### ReRanker Inference:\n\nRun QA-pair Retrieval using `paq/rerankers/rerank.py`. You can see argument help by passing `-h`. \nPass retrieval results files of the format produced by `paq/retrievers/retrieve.py` into the `--qas_to_rerank` file.\n\nIf you have many retrieval results files to rerank, it might be useful to submit them to a cluster using `submitit` to run in parallel rather than run them one by one locally.\n\nYou can pass in a comma-separated list of retrieval results filepaths to `--qas_to_rerank` (and corresponding comma-separated list of output paths to `--output_file`) to do this, and specify the number of parallel jobs to schedule uing `--n_jobs`. To run reranking locally, pass in `--n_jobs -1`\n\nAn example of reranking the top 50 retrieved QA pairs on the NQ test set, using the ALBERT-xxlarge model running locally is shown below:\n```bash\n# download resources if needed:\npython -m paq.download -v -n annotated_datasets.naturalquestions\npython -m paq.download -v -n models.rerankers.reranker_multi_xxlarge\npython -m paq.download -v -n predictions.retriever_results.multi_xlarge_nq\n\n# run reranking\npython -m paq.rerankers.rerank \\\n --model_name_or_path data/models/rerankers/reranker_multi_xxlarge \\\n --qas_to_rerank data/predictions/retriever_results/multi_xlarge_nq_test.jsonl \\\n --output_file my_reranker_results.jsonl \\\n --top_k 50 \\\n --fp16 \\\n --batch_size 4 --verbose --n_jobs -1\n```\n\n##### Evaluating Rerankers:\nEvalute the results of reranking using the `eval_reranker.py` file, this will return the Exact Match Score:\n\n```\n$ python -m paq.evaluation.eval_reranker --predictions my_reranker_results.jsonl --references data/annotated_datasets/NQ-open.test.jsonl\n47.6%\n(1699 / 3610)\n```\n\n### Question-Answer Pair Generation\n\nThe following sections details how to run the PAQ QA-Pair generation.\n\nTL;DR for users who just want to generate QA pairs: The easiest way to generate QA-pairs is to use the [End2End Generation Tool](#end2end-generation-tool) section.\n\nEach step in the pipeline can be run by itself, as described in the [Passage Scoring/Ranking](#passage-scoringranking), [Answer Extraction](#answer-extraction), [Question Generation](#question-generation) and [Filtering Generated QA-pairs](#filtering-generated-qa-pairs) section,\nor the generation pipeline can be run fully end2end (from passages to filtered QA pairs), as described in the [End2End Generation Tool](#end2end-generation-tool) section.\n\nTraining code for training your own models is coming soon.\n\nThe pipelines have a lot of configurations and options, so to keep track of these, we use json config files to specify pipeline behaviours.\nA number of example configs are listed in the `generator_configs` directory, or you can adapt them or write your own to fit your own needs.\n\n#### Passage Scoring/Ranking\n\nTo perform passage ranking, use the `paq.generation.passage_scorer.score_passages` program, which takes as input a config json file and file of passages formatted as a tsv (passage id, passage text, passage title).\n\nThere are three passage rankers implemented:\n* `DummyPassageScorer`: Applies the same score to all documents. An example config for this scorer is `generator_configs/passage_ranker_configs/dummy_passage_scorer_config.json`\n* `LookupPassageScorer`: Looks up precomputed scores based on passage id (useful if you run the same passages through the pipeline a lot, and want to save compute). An example config for this scorer is `generator_configs/passage_ranker_configs/lookup_passage_scorer_config.json`\n* `LearntPassageScorer`: Use a trained Passage Scorer (as done in the Paper). An example config for this scorer is `generator_configs/passage_ranker_configs/learnt_passage_scorer_config.json`\n\nA trained passage scorer is available for download: \n\n| Model | Training data | Architecture | Download Resource Key Name |\n| ------------- |---------- |---- | ---- |\n| passage_ranker_base| NQ | BERT-base | `models.passage_rankers.passage_ranker_base`| \n\nNote, the original Passage ranker model used in the paper was unfortunately lost due to a storage corruption issue.\nThe model available here is a reproduction using the same hardware and HPs, but differs a little due to the stochastic training sampling procedure.\n\nBelow is an example to get passage scores for the the first 1000 passages of wikipedia:\n\n```bash\n# download the passage scorer model, and wikipedia text\npython -m paq.download -v -n models.passage_rankers.passage_ranker_base\npython -m paq.download -v -n paq.psgs_w100\n\n# get 1000 passages to score\nhead -n 1000 data/paq/psgs_w100.tsv > data/paq/psgs_w100.first_1000.tsv\n\n# run scoring\npython -m paq.generation.passage_scorer.score_passages \\\n --passages_to_score data/paq/psgs_w100.first_1000.tsv \\\n --output_path my_passages_with_scores.jsonl \\\n --path_to_config generator_configs/passage_ranker_configs/learnt_passage_scorer_config.json \\\n --verbose\n\n```\n\nThis will output a jsonl file with the following format (which is accepted by the [Answer Extraction](#answer-extraction) component below)\n```json\n{\n \"passage_id\": \"ID for passage\", \n \"passage\": \"Main text of passage.\",\n \"metadata\": {\"title\": \"Title of passage\", \"ps_score\": \"passage score\"}\n}\n```\n\n#### Answer Extraction\n\nTo perform answer extraction on passages, use the `paq.generation.answer_extractor.extract_answers` program, which takes as input a config file and passages formatted in the output format of the [Passage Scoring/Ranking](#passage-scoringranking) functionality.\n\nThere are two answer extractors implemented:\n* `SpacyNERExtractor`: This answer extractor will extract named entities from passages as answers (as used in PAQ-NE). An example config for this extractor is `generator_configs/answer_extractor_configs/named_entity_answer_extractor_config.json` \n* `Span2DAnswerExtractor`: This answer extractor uses a learnt answer span extractor to extract answers (as used in PAQ-L). An example config for this extractor is `generator_configs/answer_extractor_configs/learnt_answer_extractor_config.json`\n\nThe learnt answer span extractor model used in the paper is available for download:\n\n\n| Model | Description | Training data | Architecture | Download Resource Key Name |\n| ----------| --- |---------- |---- | ---- |\n| answer_extractor_nq_base| Learnt Answer Span Extractor, BERT-base, NQ-trained | NQ | BERT-base | `models.answer_extractors.answer_extractor_nq_base`| \n\n\nBelow is an example to extract answers from passages, using the learnt extractor:\n\n```bash\n\n# download the span extractor model:\npython -m paq.download -v -n models.answer_extractors.answer_extractor_nq_base\n\n# run answer extraction\npython -m paq.generation.answer_extractor.extract_answers \\\n --passages_to_extract_from my_passages_with_scores.jsonl \\\n --output_path my_pasages_with_answers.jsonl \\\n --path_to_config generator_configs/answer_extractor_configs/learnt_answer_extractor_config.json \\\n --verbose\n```\nThis will output a jsonl file with the following format (which is accepted by the [Question Generation](#question-generation) component below)\n\n```json\n{\n \"passage_id\": \"ID for passage\", \n \"passage\": \"Main text of passage.\",\n \"metadata\": {\"title\": \"Title of passage\", \"ps_score\": \"passage score\"},\n \"answers\": [{\"text\": \"Main\", \"start\": 0, \"end\": 5, \"score\": \"score for answer\"}, {\"text\": \"passage\", \"start\": 13, \"end\": 20, \"score\": \"score for answer\"}]\n}\n```\n\n#### Question Generation\n\nTo perform Question Generation on passages with extracted answers, use the `paq.generation.question_generator.generate_questions` program, which takes as input a config file and passages with answers formatted in the output format of the [Answer Extraction](#answer-extraction) functionality. \n\nAn example config for question generation can be found here: `generator_configs/question_generator_configs/question_generation_config.json`\n\nThe following trained question generators are available:\n\n| Model | Training data | Architecture | Download Resource Key Name |\n| ------------- |---------- |---- | ---- |\n| qgen_nq_base| NQ | BART-base | `models.qgen.qgen_nq_base`| \n| qgen_multi_base| Multitask | BART-base | `models.qgen.qgen_multi_base`| \n\n\nBelow is an example to generate questions from passages with extracted answers, using the multitask generator:\n\n```bash\n\n# download the qgen model:\npython -m paq.download -v -n models.qgen.qgen_multi_base\n\n# run question generation extraction\npython -m paq.generation.question_generator.generate_questions \\\n --passage_answer_pairs_to_generate_from my_pasages_with_answers.jsonl \\\n --output_path my_generated_questions.jsonl \\\n --path_to_config generator_configs/question_generator_configs/question_generation_config.json \\\n --verbose\n```\n\nThis will output a jsonl file with the following format (which is accepted by the [Filtering Generated QA-pairs](#filtering-generated-qa-pairs) component below)\n\n```json\n{\n \"passage_id\": \"ID for passage\", \n \"answer\": \"Benedict\", \n \"question\": \"which pope has the middle name gregory\",\n \"metadata\": {\"answer_start\": 617, \"answer_end\": 625, \"ae_score\": \"score for answer\", \"qg_score\": \"currently not implemented, but score for question can go here\"}\n}\n```\n\n#### Filtering Generated QA-pairs\n\nGenerated questions can be inconsistent, or poor quality, or overly ambiguous. \nEmpirically, we find it important to filter the generated questions for answer consistency. \nTo perform filtering on generated questions, use the `paq.generation.filtering.filter_questions` program, which takes as input a config file, and generated questions formatted in the output format of the [Question Generation](#question-generation) functionality.\n\nFiltering is split into two parts: retrieval and reading. \nThe retriever retrieves passages from a corpus using the generated question, and the reader reads the passages and computes an answer.\n\nWe have implemented the following filterers:\n* *Dummy filtering*: uses a `DummyFilteringRetriever` and `DummyReader`, assigns all answers as consistent. An example config is `generator_configs/filterer_configs/dummy_filtering_config.json` \n* *Local filtering* (fast but not as good): essentially performs reading comprehension. uses a `LocalFilteringRetriever` to \"retrieve\" the passage the question was generated from. The reader (`FiDReader`) generates an answer using only this single gold passage. We use FID supplied with a single passage as the reader, which worked as well as standard readers in our experiments. An example config is `generator_configs/filterer_configs/local_filtering_config.json`.\n* *Global Filtering* (slow but important for strong performance): Uses A `GlobalFilteringRetriever` to retrieve relevant passages for the question (this uses DPR under the hood). The reader is a `FiDReader`, (this is FID under the hood). An example config is `generator_configs/filterer_configs/global_filtering_config.json`\n\nThe following trained models are available for download:\n\n| Model | Description | Training data | Architecture | Download Resource Key Name |\n| ----------| --- |---------- |---- | ---- |\n| dpr_nq_passage_retriever| DPR Passage retriever and faiss index, from the DPR Paper, used for retrieving passage for the reader in global filtering, NQ-trained| NQ | BERT-base | `models.filtering.dpr_nq_passage_retriever`| \n| fid_reader_nq_base| FID-base reader, from the Fusion-in-Decoder paper, used in global and local filtering, NQ-trained | NQ | t5-base | `models.filtering.fid_reader_nq_base`| \n\nBelow is an example of how to filter questions (both with local and global filtering):\n\n```bash\n# download the corpus to retrieve from, the DPR retriever and the reader:\npython -m paq.download -v -n paq.psgs_w100\npython -m paq.download -v -n models.filtering.dpr_nq_passage_retriever\npython -m paq.download -v -n models.filtering.fid_reader_nq_base\n\n# run filtering using local filtering...\npython -m paq.generation.filtering.filter_questions \\\n --generated_questions_to_filter my_generated_questions.jsonl \\\n --output_path my_locally_filtered_questions.jsonl \\\n --path_to_config generator_configs/filterer_configs/local_filtering_config.json \\\n --verbose\n\n# or, run filtering using global filtering \npython -m paq.generation.filtering.filter_questions \\\n --generated_questions_to_filter my_generated_questions.jsonl \\\n --output_path my_globally_filtered_questions.jsonl \\\n --path_to_config generator_configs/filterer_configs/global_filtering_config.json \\\n --verbose\n\n```\n\nThis will output a jsonl file with the following format:\n\n```json\n{\n \"passage_id\": \"ID for passage\", \n \"answer\": \"Benedict\", \n \"question\": \"which pope has the middle name gregory\",\n \"metadata\": {\"filter_answer\": \"benedict\", \"consistent\": true, \"answer_start\": 617, \"answer_end\": 625, \"ae_score\": \"score for answer\", \"qg_score\": \"currently not implemented, but score for question can go here\"}\n}\n```\n\n#### End2End Generation Tool\n\nTo run all the steps in the pipeline end2end, use the `paq.generation.generate_qa_pairs` program.\nThis will run passage ranking, then answer extraction, then generation, then finally filtering automatically.\n\nThe tool takes as input a config json file, and a file passages to generate QA pairs from, formatted as a tsv (passage id, passage text, passage title).\nThe tool will create out output directory, and write intermediate results to it, including the final generated QA-pairs in the `final_qas.jsonl` file.\n\nThe following example configs can be used with this tool to replicate the generation pipelines used in the paper:\n* `generator_configs/paq_L1_config.json`: run a generation pipeline replicating PAQ-L1\n* `generator_configs/paq_L4_config.json`: run a generation pipeline replicating PAQ-L4\n* `generator_configs/paq_NE_config.json`: run a generation pipeline replicating PAQ-NE\n* `generator_configs/paq_L1_local_filtering_config.json`: run a generation pipeline replicating PAQ-L1, but with local rather than global filtering.\n\nOr, write your own config to fit your generation needs.\n\nThe following code will run the PAQ-L1 generation pipeline on the first 1000 passages in the preprocssed wikipedia dump:\n\n```bash\n# Download the models and data we need:\npython -m paq.download -v -n models.passage_rankers.passage_ranker_base\npython -m paq.download -v -n models.answer_extractors.answer_extractor_nq_base\npython -m paq.download -v -n models.qgen.qgen_multi_base\npython -m paq.download -v -n paq.psgs_w100\npython -m paq.download -v -n models.filtering.dpr_nq_passage_retriever\npython -m paq.download -v -n models.filtering.fid_reader_nq_base\n\nhead -n 1000 data/paq/psgs_w100.tsv > data/paq/psgs_w100.first_1000.tsv\n\npython -m paq.generation.generate_qa_pairs \\\n --passage_files_to_generate data/paq/psgs_w100.first_1000.tsv \\\n --output_dirs my_generated_qas \\\n --path_to_config generator_configs/paq_L1_config.json\\\n --verbose --n_jobs -1\n```\n\nThe paq.generation.generate_qa_pairs can use submitit to run generation on a cluster.\nThe `--n_jobs` flag indicates how many concurrent submitit jobs to submit, use --n_jobs -1 to run locally.\nTo run generation in several jobs in parallel, you can pass in a comma-separated list of input files to `--passage_files_to_generate`\n and a corresponding comma separated list of output directories to create.\n\n\n\n## Citing\n\nTo cite this work, please use the following bibtex:\n```\n@article{lewis2021paq,\n title={PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them}, \n author={Patrick Lewis and Yuxiang Wu and Linqing Liu and Pasquale Minervini and Heinrich Küttler and Aleksandra Piktus and Pontus Stenetorp and Sebastian Riedel},\n year={2021},\n eprint={2102.07033},\n archivePrefix={arXiv},\n primaryClass={cs.CL}\n}\n```\n\n## LICENSE\n\n### Code License:\n\nThe majority of the PAQ code is licensed under [CC-BY-NC](./LICENSE), however portions of the project are available under separate license terms: HuggingFace Transformers is licensed under Apache License 2.0; spaCy and wandb are licensed under the MIT License.\nThe code in this repository is licenced according the [LICENSE](./LICENSE) file.\n\n### Data License:\n\nThe PAQ QA-pairs and metadata is licensed under [CC-BY-SA](https://creativecommons.org/licenses/by-sa/3.0/). \nOther data is licensed according to the accompanying license files.\n" }, { "path": "full_models_list.md", "content": "# Full List of Models Available for Download\n\n## BiEncoder Retrievers\n\n\n| Model | Training data | Architecture | Embedding Dim | NQ EM | + rerank | TQA EM | + rerank | Download Resource Key Name |\n| ------------- |----------| --- | --------- | ---------- |---- |---- | ---- | ---- |\n| retriever_multi_base_256 (recommended)| NQ + TQA | AlBERT-base | 256 | 41.4 | 47.3 | 40.2 | 50.9| `models.retrievers.retriever_multi_base_256` |\n| retriever_multi_base | NQ + TQA | AlBERT-base | 728 | 40.9| 47.4 | 39.7 | 51.2 | `models.retrievers.retriever_multi_base` |\n| retriever_multi_large | NQ + TQA | AlBERT-large | 728 | 41.2 | 47.5 | 41.0| 51.9 |`models.retrievers.retriever_multi_large`|\n| retriever_multi_xlarge | NQ + TQA | AlBERT-xlarge| 728 | 41.7 | 47.6 | 41.3 | 52.1 |`models.retrievers.retriever_multi_xlarge`|\n| retriever_nq_base | NQ | AlBERT-base | 728 | 41.0 | 47.2 |35.6 | 49.0 |`models.retrievers.retriever_nq_base`|\n| retriever_nq_large | NQ | AlBERT-large | 728 | 40.4 | 47.3| 34.1|48.1 |`models.retrievers.retriever_nq_large`|\n| retriever_nq_xlarge | NQ | AlBERT-xlarge | 728 | 41.1 |47.7 | 35.7| 48.9|`models.retrievers.retriever_nq_xlarge`|\n| retriever_tqa_base | TQA | AlBERT-base | 728 | 37.5| 46.8 | 38.7| 51.0| `models.retrievers.retriever_tqa_base`|\n| retriever_tqa_large | TQA | AlBERT-large | 728 | 38.2| 47.0| 39.6|51.4 |`models.retrievers.retriever_tqa_large`|\n| retriever_tqa_xlarge | TQA | AlBERT-xlarge | 728 | 38.0| 46.5 | 38.9|51.2 |`models.retrievers.retriever_tqa_xlarge`|\n\n(Rerank scores calculated with `reranker_multi_xxlarge`)\n\n## QA Rerankers\n\n\n| Model | Training data | Architecture | NQ EM | TQA EM | Download Resource Key Name |\n| ------------- |----------| --- | --------- | ---------- |---- |\n|reranker_multi_base| NQ + TQA| AlBERT-base |46.0 |48.9 | `models.rerankers.reranker_multi_base`| \n|reranker_multi_large| NQ + TQA|AlBERT-large | 46.2| 49.4|`models.rerankers.reranker_multi_large`| \n|reranker_multi_xlarge| NQ + TQA|AlBERT-xlarge | 46.0| 49.1| `models.rerankers.reranker_multi_xlarge`| \n|reranker_multi_xxlarge| NQ + TQA|AlBERT-xxlarge | 47.7| 52.1 | `models.rerankers.reranker_multi_xxlarge`| \n|reranker_nq_xlarge| NQ | AlBERT-xlarge | 45.2| 46.7 | `models.rerankers.reranker_nq_xlarge`| \n|reranker_nq_xxlarge| NQ| AlBERT-xxlarge |46.4 | 49.6| `models.rerankers.reranker_nq_xxlarge`| \n|reranker_tqa_xlarge| TQA | AlBERT-xlarge | 45.0|49.7 | `models.rerankers.reranker_tqa_xlarge`| \n|reranker_tqa_xxlarge| TQA | AlBERT-xxlarge | 46.0|51.7 | `models.rerankers.reranker_tqa_xxlarge`| \n\n(EM scores in this table calculated using `retriever_multi_xlarge` retriever)\n\n## Qgen Models\n\n| Model | Training data | Architecture | Download Resource Key Name |\n| ------------- |---------- |---- | ---- |\n| qgen_nq_base| NQ | BART-base | `models.qgen.qgen_nq_base`| \n| qgen_multi_base| Multitask | BART-base | `models.qgen.qgen_multi_base`| \n\n\n## Passage Ranker Models\n\nModels used for selecting passages to generate questions from:\n\n| Model | Training data | Architecture | Download Resource Key Name |\n| ------------- |---------- |---- | ---- |\n| passage_ranker_base| NQ | BERT-base | `models.passage_rankers.passage_ranker_base`| \n\nNote, the original Passage ranker model used in the paper was unfortunately lost due to a storage corruption issue.\nThe model here is a reproduction using the same hardware and HPs, but differs a little due to the stochastic training sampling procedure.\n\n## Answer Extractor Models\n\n\n| Model | Description | Training data | Architecture | Download Resource Key Name |\n| ----------| --- |---------- |---- | ---- |\n| answer_extractor_nq_base| Learnt Answer Span Extractor, BERT-base, NQ-trained | NQ | BERT-base | `models.answer_extractors.answer_extractor_nq_base`| \n\n## Filterer Models\n\n| Model | Description | Training data | Architecture | Download Resource Key Name |\n| ----------| --- |---------- |---- | ---- |\n| dpr_nq_passage_retriever| DPR Passage retriever and faiss index, from the DPR Paper, used for retrieving passage for the reader in global filtering, NQ-trained| NQ | BERT-base | `models.filtering.dpr_nq_passage_retriever`| \n| fid_reader_nq_base| FID-base reader, from the Fusion-in-Decoder paper, used in global and local filtering, NQ-trained | NQ | t5-base | `models.filtering.fid_reader_nq_base`| \n" }, { "path": "generator_configs/answer_extractor_configs/learnt_answer_extractor_config.json", "content": "{\n \"answer_extractor\": {\n \"name\": \"answer_extractor/span2D\",\n \"config\": {\n \"model_path\": \"data/models/answer_extractors/answer_extractor_nq_base\",\n \"config_path\": \"data/models/answer_extractors/answer_extractor_nq_base\",\n \"tokenizer_path\": \"data/models/answer_extractors/answer_extractor_nq_base\",\n \"topk\": 8,\n \"max_answer_len\": 30,\n \"max_seq_len\": 256,\n \"doc_stride\": 128,\n \"batch_size\": 128,\n \"device\": 0\n }\n }\n}" }, { "path": "generator_configs/answer_extractor_configs/named_entity_answer_extractor_config.json", "content": "{\n \"answer_extractor\": {\n \"name\": \"answer_extractor/spacy_ner\",\n \"config\": {\n \"model\": \"en_core_web_sm\"\n }\n }\n}" }, { "path": "generator_configs/filterer_configs/dummy_filtering_config.json", "content": "{\n \"filterer\": {\n \"retriever\": {\n \"name\": \"filtering/dummy_filtering_retriever\",\n \"config\": {\n }\n },\n \"reader\": {\n \"name\": \"filtering/dummy_reader\",\n \"config\": {\n }\n }\n }\n}" }, { "path": "generator_configs/filterer_configs/global_filtering_config.json", "content": "{\n \"filterer\": {\n \"retriever\": {\n \"name\": \"filtering/global_filtering_retriever\",\n \"config\": {\n \"corpus_path\": \"data/paq/psgs_w100.tsv\",\n \"index_path\": \"data/models/filtering/dpr_nq_passage_retriever/dpr_index.hnsw.SQ8.index.dpr\",\n \"index_id_to_db_id_path\": \"data/models/filtering/dpr_nq_passage_retriever/dpr_index.hnsw.SQ8.index_meta.dpr\",\n \"model_path\": \"data/models/filtering/dpr_nq_passage_retriever\",\n \"batch_size\": 128,\n \"n_queries_to_parallelize\": 2048,\n \"max_seq_len\":256,\n \"n_docs\": 50,\n \"device\": 0\n }\n },\n \"reader\": {\n \"name\": \"filtering/fid_reader\",\n \"config\": {\n \"model_path\": \"data/models/filtering/fid_reader_nq_base\",\n \"batch_size\": 4,\n \"device\": 0,\n \"max_seq_len\": 200,\n \"n_docs\": 50\n }\n }\n }\n}" }, { "path": "generator_configs/filterer_configs/local_filtering_config.json", "content": "{\n \"filterer\": {\n \"retriever\": {\n \"name\": \"filtering/local_filtering_retriever\",\n \"config\": {\n \"corpus_path\": \"data/paq/psgs_w100.tsv\"\n }\n },\n \"reader\": {\n \"name\": \"filtering/fid_reader\",\n \"config\": {\n \"model_path\": \"data/models/filtering/fid_reader_nq_base\",\n \"batch_size\": 32,\n \"device\": 0,\n \"max_seq_len\": 200,\n \"n_docs\": 1\n }\n }\n }\n}" }, { "path": "generator_configs/paq_L1_config.json", "content": "{\n \"passage_scorer\": {\n \"name\": \"passage_scorer/learnt\",\n \"config\": {\n \"model_path\":\"data/models/passage_rankers/passage_ranker_base\",\n \"config_path\":\"data/models/passage_rankers/passage_ranker_base\",\n \"tokenizer_path\":\"data/models/passage_rankers/passage_ranker_base\",\n \"device\": 0,\n \"batch_size\": 64,\n \"max_seq_len\": 256\n }\n },\n \"answer_extractor\": {\n \"name\": \"answer_extractor/span2D\",\n \"config\": {\n \"model_path\": \"data/models/answer_extractors/answer_extractor_nq_base\",\n \"config_path\": \"data/models/answer_extractors/answer_extractor_nq_base\",\n \"tokenizer_path\": \"data/models/answer_extractors/answer_extractor_nq_base\",\n \"topk\": 8,\n \"max_answer_len\": 30,\n \"max_seq_len\": 256,\n \"doc_stride\": 128,\n \"batch_size\": 128,\n \"device\": 0\n }\n },\n \"question_generator\": {\n \"name\": \"question_generator/standard\",\n \"config\": {\n \"model_path\": \"data/models/qgen/qgen_multi_base\",\n \"config_path\": null,\n \"tokenizer_path\": \"data/models/qgen/qgen_multi_base\",\n \"include_title\": true,\n \"num_beams\": 4,\n \"num_return_sequences\": 1,\n \"max_question_len\": 20,\n \"batch_size\": 64,\n \"device\": 0\n }\n },\n \"filterer\": {\n \"retriever\": {\n \"name\": \"filtering/global_filtering_retriever\",\n \"config\": {\n \"corpus_path\": \"data/paq/psgs_w100.tsv\",\n \"index_path\": \"data/models/filtering/dpr_nq_passage_retriever/dpr_index.hnsw.SQ8.index.dpr\",\n \"index_id_to_db_id_path\": \"data/models/filtering/dpr_nq_passage_retriever/dpr_index.hnsw.SQ8.index_meta.dpr\",\n \"model_path\": \"data/models/filtering/dpr_nq_passage_retriever\",\n \"batch_size\": 128,\n \"n_queries_to_parallelize\": 2048,\n \"max_seq_len\":256,\n \"n_docs\": 50,\n \"device\": 0\n }\n },\n \"reader\": {\n \"name\": \"filtering/fid_reader\",\n \"config\": {\n \"model_path\": \"data/models/filtering/fid_reader_nq_base\",\n \"batch_size\": 4,\n \"device\": 0,\n \"max_seq_len\": 200,\n \"n_docs\": 50\n }\n }\n }\n}" }, { "path": "generator_configs/paq_L1_with_local_filtering_config.json", "content": "{\n \"passage_scorer\": {\n \"name\": \"passage_scorer/learnt\",\n \"config\": {\n \"model_path\":\"data/models/passage_rankers/passage_ranker_base\",\n \"config_path\":\"data/models/passage_rankers/passage_ranker_base\",\n \"tokenizer_path\":\"data/models/passage_rankers/passage_ranker_base\",\n \"device\": 0,\n \"batch_size\": 64,\n \"max_seq_len\": 256\n }\n },\n \"answer_extractor\": {\n \"name\": \"answer_extractor/span2D\",\n \"config\": {\n \"model_path\": \"data/models/answer_extractors/answer_extractor_nq_base\",\n \"config_path\": \"data/models/answer_extractors/answer_extractor_nq_base\",\n \"tokenizer_path\": \"data/models/answer_extractors/answer_extractor_nq_base\",\n \"topk\": 8,\n \"max_answer_len\": 30,\n \"max_seq_len\": 256,\n \"doc_stride\": 128,\n \"batch_size\": 128,\n \"device\": 0\n }\n },\n \"question_generator\": {\n \"name\": \"question_generator/standard\",\n \"config\": {\n \"model_path\": \"data/models/qgen/qgen_multi_base\",\n \"config_path\": null,\n \"tokenizer_path\": \"data/models/qgen/qgen_multi_base\",\n \"include_title\": true,\n \"num_beams\": 4,\n \"num_return_sequences\": 1,\n \"max_question_len\": 20,\n \"batch_size\": 64,\n \"device\": 0\n }\n },\n \"filterer\": {\n \"retriever\": {\n \"name\": \"filtering/local_filtering_retriever\",\n \"config\": {\n \"corpus_path\": \"data/paq/psgs_w100.tsv\"\n }\n },\n \"reader\": {\n \"name\": \"filtering/fid_reader\",\n \"config\": {\n \"model_path\": \"data/models/filtering/fid_reader_nq_base\",\n \"batch_size\": 32,\n \"device\": 0,\n \"max_seq_len\": 200,\n \"n_docs\": 1\n }\n }\n }\n}" }, { "path": "generator_configs/paq_L4_config.json", "content": "{\n \"passage_scorer\": {\n \"name\": \"passage_scorer/learnt\",\n \"config\": {\n \"model_path\":\"data/models/passage_rankers/passage_ranker_base\",\n \"config_path\":\"data/models/passage_rankers/passage_ranker_base\",\n \"tokenizer_path\":\"data/models/passage_rankers/passage_ranker_base\",\n \"device\": 0,\n \"batch_size\": 64,\n \"max_seq_len\": 256\n }\n },\n \"answer_extractor\": {\n \"name\": \"answer_extractor/span2D\",\n \"config\": {\n \"model_path\": \"data/models/answer_extractors/answer_extractor_nq_base\",\n \"config_path\": \"data/models/answer_extractors/answer_extractor_nq_base\",\n \"tokenizer_path\": \"data/models/answer_extractors/answer_extractor_nq_base\",\n \"topk\": 8,\n \"max_answer_len\": 30,\n \"max_seq_len\": 256,\n \"doc_stride\": 128,\n \"batch_size\": 128,\n \"device\": 0\n }\n },\n \"question_generator\": {\n \"name\": \"question_generator/standard\",\n \"config\": {\n \"model_path\": \"data/models/qgen/qgen_multi_base\",\n \"config_path\": null,\n \"tokenizer_path\": \"data/models/qgen/qgen_multi_base\",\n \"include_title\": true,\n \"num_beams\": 4,\n \"num_return_sequences\": 4,\n \"max_question_len\": 20,\n \"batch_size\": 64,\n \"device\": 0\n }\n },\n \"filterer\": {\n \"retriever\": {\n \"name\": \"filtering/global_filtering_retriever\",\n \"config\": {\n \"corpus_path\": \"data/paq/psgs_w100.tsv\",\n \"index_path\": \"data/models/filtering/dpr_nq_passage_retriever/dpr_index.hnsw.SQ8.index.dpr\",\n \"index_id_to_db_id_path\": \"data/models/filtering/dpr_nq_passage_retriever/dpr_index.hnsw.SQ8.index_meta.dpr\",\n \"model_path\": \"data/models/filtering/dpr_nq_passage_retriever\",\n \"batch_size\": 128,\n \"n_queries_to_parallelize\": 2048,\n \"max_seq_len\":256,\n \"n_docs\": 50,\n \"device\": 0\n }\n },\n \"reader\": {\n \"name\": \"filtering/fid_reader\",\n \"config\": {\n \"model_path\": \"data/models/filtering/fid_reader_nq_base\",\n \"batch_size\": 4,\n \"device\": 0,\n \"max_seq_len\": 200,\n \"n_docs\": 50\n }\n }\n }\n}" }, { "path": "generator_configs/paq_NE_config.json", "content": "{\n \"passage_scorer\": {\n \"name\": \"passage_scorer/learnt\",\n \"config\": {\n \"model_path\":\"data/models/passage_rankers/passage_ranker_base\",\n \"config_path\":\"data/models/passage_rankers/passage_ranker_base\",\n \"tokenizer_path\":\"data/models/passage_rankers/passage_ranker_base\",\n \"device\": 0,\n \"batch_size\": 64,\n \"max_seq_len\": 256\n }\n },\n \"answer_extractor\": {\n \"name\": \"answer_extractor/spacy_ner\",\n \"config\": {\n \"model\": \"en_core_web_sm\"\n }\n },\n \"question_generator\": {\n \"name\": \"question_generator/standard\",\n \"config\": {\n \"model_path\": \"data/models/qgen/qgen_nq_base\",\n \"config_path\": null,\n \"tokenizer_path\": \"data/models/qgen/qgen_nq_base\",\n \"include_title\": true,\n \"num_beams\": 4,\n \"num_return_sequences\": 1,\n \"max_question_len\": 20,\n \"batch_size\": 64,\n \"device\": 0\n }\n },\n \"filterer\": {\n \"retriever\": {\n \"name\": \"filtering/global_filtering_retriever\",\n \"config\": {\n \"corpus_path\": \"data/paq/psgs_w100.tsv\",\n \"index_path\": \"data/models/filtering/dpr_nq_passage_retriever/dpr_index.hnsw.SQ8.index.dpr\",\n \"index_id_to_db_id_path\": \"data/models/filtering/dpr_nq_passage_retriever/dpr_index.hnsw.SQ8.index_meta.dpr\",\n \"model_path\": \"data/models/filtering/dpr_nq_passage_retriever\",\n \"batch_size\": 128,\n \"n_queries_to_parallelize\": 2048,\n \"max_seq_len\":256,\n \"n_docs\": 50,\n \"device\": 0\n }\n },\n \"reader\": {\n \"name\": \"filtering/fid_reader\",\n \"config\": {\n \"model_path\": \"data/models/filtering/fid_reader_nq_base\",\n \"batch_size\": 4,\n \"device\": 0,\n \"max_seq_len\": 200,\n \"n_docs\": 50\n }\n }\n }\n}" }, { "path": "generator_configs/passage_ranker_configs/dummy_passage_scorer_config.json", "content": "{\n \"passage_scorer\": {\n \"name\": \"passage_scorer/dummy\",\n \"config\":{\n \"default_score\": -1000\n }\n }\n}" }, { "path": "generator_configs/passage_ranker_configs/learnt_passage_scorer_config.json", "content": "{\n \"passage_scorer\": {\n \"name\": \"passage_scorer/learnt\",\n \"config\": {\n \"model_path\":\"data/models/passage_rankers/passage_ranker_base\",\n \"config_path\":\"data/models/passage_rankers/passage_ranker_base\",\n \"tokenizer_path\":\"data/models/passage_rankers/passage_ranker_base\",\n \"device\": 0,\n \"batch_size\": 64,\n \"max_seq_len\": 256\n }\n }\n}" }, { "path": "generator_configs/passage_ranker_configs/lookup_passage_scorer_config.json", "content": "{\n \"passage_scorer\": {\n \"name\": \"passage_scorer/lookup\",\n \"config\":{\n \"default_score\": -1000,\n \"scores_file\": \"data/paq/PASSAGE_SCORES/passage_scores.tsv\"\n }\n }\n}" }, { "path": "generator_configs/question_generator_configs/question_generation_config.json", "content": "{\n \"question_generator\": {\n \"name\": \"question_generator/standard\",\n \"config\": {\n \"model_path\": \"data/models/qgen/qgen_multi_base\",\n \"config_path\": null,\n \"tokenizer_path\": \"data/models/qgen/qgen_multi_base\",\n \"include_title\": true,\n \"num_beams\": 4,\n \"num_return_sequences\": 1,\n \"max_question_len\": 20,\n \"batch_size\": 64,\n \"device\": 0\n }\n }\n}" }, { "path": "paq/__init__.py", "content": "#!/usr/bin/env python3\n# Copyright (c) Facebook, Inc. and its affiliates.\n# All rights reserved.\n#\n# This source code is licensed under the license found in the\n# LICENSE file in the root directory of this source tree." }, { "path": "paq/download.py", "content": "#!/usr/bin/env python3\n# Copyright (c) Facebook, Inc. and its affiliates.\n# All rights reserved.\n#\n# This source code is licensed under the license found in the\n# LICENSE file in the root directory of this source tree.\nimport argparse\nimport gzip\nimport logging\nimport os\nimport pathlib\nimport wget\nimport tarfile\n\nfrom typing import Tuple, List\n\nlogger = logging.getLogger(__name__)\n\n\nNQ_LICENSE_FILES = [\n \"https://dl.fbaipublicfiles.com/dpr/nq_license/LICENSE\",\n \"https://dl.fbaipublicfiles.com/dpr/nq_license/README\",\n]\n\nRESOURCES_MAP = {\n \"paq.PAQ\": {\n 's3_url': 'https://dl.fbaipublicfiles.com/paq/v1/PAQ.tar.gz',\n \"original_ext\": \".tar.gz\",\n \"compressed\": True,\n \"desc\": \"Full PAQ generated QA pairs (PAQ-L + PAQ-NE)\",\n \"skip_if_exists_path\": \"paq/PAQ\"\n },\n \"paq.PAQ_L1\": {\n 's3_url': 'https://dl.fbaipublicfiles.com/paq/v1/PAQ_L1.tar.gz',\n \"original_ext\": \".tar.gz\",\n \"compressed\": True,\n \"desc\": \"PAQ-L1 subset of PAQ generated QA pairs\",\n \"skip_if_exists_path\": \"paq/PAQ_L1\"\n },\n \"paq.PAQ_L4\": {\n 's3_url': 'https://dl.fbaipublicfiles.com/paq/v1/PAQ_L4.tar.gz',\n \"original_ext\": \".tar.gz\",\n \"compressed\": True,\n \"desc\": \"PAQ-L4 subset of PAQ generated QA pairs\",\n \"skip_if_exists_path\": \"paq/PAQ_L4\"\n },\n \"paq.PAQ_NE1\": {\n 's3_url': 'https://dl.fbaipublicfiles.com/paq/v1/PAQ_NE1.tar.gz',\n \"original_ext\": \".tar.gz\",\n \"compressed\": True,\n \"desc\": \"PAQ-NE1 subset of PAQ generated QA pairs\",\n \"skip_if_exists_path\": \"paq/PAQ_NE1\"\n },\n \"paq.TQA_TRAIN_NQ_TRAIN_PAQ\": {\n 's3_url': 'https://dl.fbaipublicfiles.com/paq/v1/TQA_TRAIN_NQ_TRAIN_PAQ.tar.gz',\n \"original_ext\": \".tar.gz\",\n \"compressed\": True,\n \"desc\": \"TriviaQA train set QA pairs, NQ train set QA pairs and Full PAQ generated QA pairs\",\n \"skip_if_exists_path\": \"paq/TQA_TRAIN_NQ_TRAIN_PAQ\"\n\n },\n \"paq.psgs_w100\": {\n 's3_url': 'https://dl.fbaipublicfiles.com/paq/v1/psgs_w100.tsv.gz',\n \"original_ext\": \".tsv\",\n \"compressed\": True,\n \"desc\": \"Preprocessed wikipedia dump, split into 100 word passages\",\n \"skip_if_exists_path\": \"paq/psgs_w100.tsv\"\n },\n \"paq.PASSAGE_SCORES\": {\n 's3_url': 'https://dl.fbaipublicfiles.com/paq/v1/PASSAGE_SCORES.tar.gz',\n \"original_ext\": \".tar.gz\",\n \"compressed\": True,\n \"desc\": \"Passage selection scores for the passages in `psgs_w100`\",\n \"skip_if_exists_path\": \"paq/PASSAGE_SCORES\"\n },\n \"paq.PAQ_metadata\": {\n 's3_url': 'https://dl.fbaipublicfiles.com/paq/v1/PAQ.metadata.jsonl.gz',\n \"original_ext\": \".jsonl\",\n \"compressed\": True,\n \"desc\": \"PAQ QA pairs metadata \",\n \"skip_if_exists_path\": \"paq/PAQ.metadata.jsonl\"\n },\n \"paq.PAQ_unfiltered_metadata\": {\n 's3_url': 'https://dl.fbaipublicfiles.com/paq/v1/PAQ.unfiltered_metadata.jsonl.gz',\n \"original_ext\": \".jsonl\",\n \"compressed\": True,\n \"desc\": \"PAQ QA pairs metadata for unfiltered QA pairs\",\n \"skip_if_exists_path\": \"paq/PAQ.unfiltered_metadata.jsonl\"\n },\n\n 'annotated_datasets.naturalquestions': {\n 's3_url': [\n 'https://dl.fbaipublicfiles.com/paq/v1/annotated_datasets/NQ-open.train-train.jsonl',\n 'https://dl.fbaipublicfiles.com/paq/v1/annotated_datasets/NQ-open.train-dev.jsonl',\n 'https://dl.fbaipublicfiles.com/paq/v1/annotated_datasets/NQ-open.test.jsonl',\n 'https://dl.fbaipublicfiles.com/paq/v1/annotated_datasets/NQ_LICENSE',\n 'https://dl.fbaipublicfiles.com/paq/v1/annotated_datasets/NQ_README',\n ],\n \"original_ext\": [\".jsonl\", \".jsonl\", \".jsonl\", \"\", \"\"],\n \"compressed\": False,\n \"desc\": \"The Open NaturalQuestions QA Pairs used in our experiments\",\n \"skip_if_exists_path\": \"annotated_datasets/naturalquestions\"\n },\n 'annotated_datasets.triviaqa': {\n 's3_url': [\n 'https://dl.fbaipublicfiles.com/paq/v1/annotated_datasets/triviaqa.train-train.jsonl',\n 'https://dl.fbaipublicfiles.com/paq/v1/annotated_datasets/triviaqa.train-dev.jsonl',\n 'https://dl.fbaipublicfiles.com/paq/v1/annotated_datasets/triviaqa.test.jsonl'\n ],\n \"original_ext\": \".jsonl\",\n \"compressed\": False,\n \"desc\": \"The TriviaQA QA Pairs used in our experiments\",\n \"skip_if_exists_path\": \"annotated_datasets/triviaqa\"\n },\n\n \"models.retrievers.retriever_multi_base_256\": {\n 's3_url': 'https://dl.fbaipublicfiles.com/paq/v1/models/retrievers/retriever_multi_base_256.tar.gz',\n \"original_ext\": \".tar.gz\",\n \"compressed\": True,\n \"desc\": \"RePAQ Retriever Albert-Base model with 256 output embedding dim, multask. Recommended RePAQ retriever\",\n \"skip_if_exists_path\": \"models/retrievers/retriever_multi_base_256\"\n },\n \"models.retrievers.retriever_multi_base\": {\n 's3_url': 'https://dl.fbaipublicfiles.com/paq/v1/models/retrievers/retriever_multi_base.tar.gz',\n \"original_ext\": \".tar.gz\",\n \"compressed\": True,\n \"desc\": \"RePAQ Retriever Albert-Base model with 768 output embedding dim, multitask\",\n \"skip_if_exists_path\": \"models/retrievers/retriever_multi_base\"\n },\n \"models.retrievers.retriever_multi_large\": {\n 's3_url': 'https://dl.fbaipublicfiles.com/paq/v1/models/retrievers/retriever_multi_large.tar.gz',\n \"original_ext\": \".tar.gz\",\n \"compressed\": True,\n \"desc\": \"RePAQ Retriever Albert-Large model with 768 output embedding dim, multitask\",\n \"skip_if_exists_path\": \"models/retrievers/retriever_multi_large\"\n },\n \"models.retrievers.retriever_multi_xlarge\": {\n 's3_url': 'https://dl.fbaipublicfiles.com/paq/v1/models/retrievers/retriever_multi_xlarge.tar.gz',\n \"original_ext\": \".tar.gz\",\n \"compressed\": True,\n \"desc\": \"RePAQ Retriever Albert-xlarge model with 768 output embedding dim, multitask\",\n \"skip_if_exists_path\": \"models/retrievers/retriever_multi_xlarge\"\n },\n \"models.retrievers.retriever_nq_base\": {\n 's3_url': 'https://dl.fbaipublicfiles.com/paq/v1/models/retrievers/retriever_nq_base.tar.gz',\n \"original_ext\": \".tar.gz\",\n \"compressed\": True,\n \"desc\": \"RePAQ Retriever Albert-base model with 768 output embedding dim, trained on NQ\",\n \"skip_if_exists_path\": \"models/retrievers/retriever_nq_base\"\n },\n \"models.retrievers.retriever_nq_large\": {\n 's3_url': 'https://dl.fbaipublicfiles.com/paq/v1/models/retrievers/retriever_nq_large.tar.gz',\n \"original_ext\": \".tar.gz\",\n \"compressed\": True,\n \"desc\": \"RePAQ Retriever Albert-large model with 768 output embedding dim, trained on NQ\",\n \"skip_if_exists_path\": \"models/retrievers/retriever_nq_large\"\n },\n \"models.retrievers.retriever_nq_xlarge\": {\n 's3_url': 'https://dl.fbaipublicfiles.com/paq/v1/models/retrievers/retriever_nq_xlarge.tar.gz',\n \"original_ext\": \".tar.gz\",\n \"compressed\": True,\n \"desc\": \"RePAQ Retriever Albert-xlarge model with 768 output embedding dim, trained on NQ\",\n \"skip_if_exists_path\": \"models/retrievers/retriever_nq_xlarge\"\n\n },\n \"models.retrievers.retriever_tqa_base\": {\n 's3_url': 'https://dl.fbaipublicfiles.com/paq/v1/models/retrievers/retriever_tqa_base.tar.gz',\n \"original_ext\": \".tar.gz\",\n \"compressed\": True,\n \"desc\": \"RePAQ Retriever Albert-base model with 768 output embedding dim, trained on TriviaQA\",\n \"skip_if_exists_path\": \"models/retrievers/retriever_tqa_base\"\n },\n \"models.retrievers.retriever_tqa_large\": {\n 's3_url': 'https://dl.fbaipublicfiles.com/paq/v1/models/retrievers/retriever_tqa_large.tar.gz',\n \"original_ext\": \".tar.gz\",\n \"compressed\": True,\n \"desc\": \"RePAQ Retriever Albert-large model with 768 output embedding dim, trained on TriviaQA\",\n \"skip_if_exists_path\": \"models/retrievers/retriever_tqa_large\"\n },\n \"models.retrievers.retriever_tqa_xlarge\": {\n 's3_url': 'https://dl.fbaipublicfiles.com/paq/v1/models/retrievers/retriever_tqa_xlarge.tar.gz',\n \"original_ext\": \".tar.gz\",\n \"compressed\": True,\n \"desc\": \"RePAQ Retriever Albert-xlarge model with 768 output embedding dim, trained on TriviaQA\",\n \"skip_if_exists_path\": \"models/retrievers/retriever_tqa_xlarge\"\n },\n\n\n\n 'vectors.multi_base_256_vectors': {\n \"s3_url\": [\n \"https://dl.fbaipublicfiles.com/paq/v1/models/vectors/multi_base_256_vectors/embeddings.pt.{}\".format(\n i\n )\n for i in range(50)\n ],\n \"original_ext\": \".pt\",\n \"compressed\": False,\n \"desc\": \"Precomputed vectors for tqa-train-nq-train-PAQ.jsonl, using the `multi_base_256` model\",\n \"skip_if_exists_path\": \"vectors/multi_base_256_vectors\"\n },\n\n\n \"indices.multi_base_256_flat_sq8\": {\n 's3_url': 'https://dl.fbaipublicfiles.com/paq/v1/models/indices/multi_base_256.flat.sq8.faiss',\n \"original_ext\": \".faiss\",\n \"compressed\": False,\n \"desc\": \"Precomputed Flat Faiss Index for tqa-train-nq-train-PAQ.jsonl, using the `multi_base_256` model. Slow but exact\",\n \"skip_if_exists_path\": \"indices/multi_base_256_flat_sq8\"\n },\n \"indices.multi_base_256_hnsw_sq8\": {\n 's3_url': 'https://dl.fbaipublicfiles.com/paq/v1/models/indices/multi_base_256.hnsw.sq8.faiss',\n \"original_ext\": \".faiss\",\n \"compressed\": False,\n \"desc\": \"Precomputed Flat Faiss Index for tqa-train-nq-train-PAQ.jsonl, using the `multi_base_256` model. \"\n \"Very Fast but slightly less accurate than `multi_base_256_flat_sq8`\",\n \"skip_if_exists_path\": \"indices/multi_base_256_hnsw_sq8\"\n },\n\n \"models.rerankers.reranker_multi_base\": {\n 's3_url': 'https://dl.fbaipublicfiles.com/paq/v1/models/rerankers/reranker_multi_base.tar.gz',\n \"original_ext\": \".tar.gz\",\n \"compressed\": True,\n \"desc\": \"RePAQ Reranker AlBERT-Base model, multitask\",\n \"skip_if_exists_path\": \"models/rerankers/reranker_multi_base\"\n\n },\n \"models.rerankers.reranker_multi_large\": {\n 's3_url': 'https://dl.fbaipublicfiles.com/paq/v1/models/rerankers/reranker_multi_large.tar.gz',\n \"original_ext\": \".tar.gz\",\n \"compressed\": True,\n \"desc\": \"RePAQ Reranker AlBERT-Large model, multitask\",\n \"skip_if_exists_path\": \"models/rerankers/reranker_multi_large\"\n },\n \"models.rerankers.reranker_multi_xlarge\": {\n 's3_url': 'https://dl.fbaipublicfiles.com/paq/v1/models/rerankers/reranker_multi_xlarge.tar.gz',\n \"original_ext\": \".tar.gz\",\n \"compressed\": True,\n \"desc\": \"RePAQ Reranker AlBERT-xlarge model, multitask\",\n \"skip_if_exists_path\": \"models/rerankers/reranker_multi_xlarge\"\n },\n \"models.rerankers.reranker_multi_xxlarge\": {\n 's3_url': 'https://dl.fbaipublicfiles.com/paq/v1/models/rerankers/reranker_multi_xxlarge.tar.gz',\n \"original_ext\": \".tar.gz\",\n \"compressed\": True,\n \"desc\": \"RePAQ Reranker AlBERT-xxlarge model, multitask\",\n \"skip_if_exists_path\": \"models/rerankers/reranker_multi_xxlarge\"\n },\n \"models.rerankers.reranker_tqa_xlarge\": {\n 's3_url': 'https://dl.fbaipublicfiles.com/paq/v1/models/rerankers/reranker_tqa_xlarge.tar.gz',\n \"original_ext\": \".tar.gz\",\n \"compressed\": True,\n \"desc\": \"RePAQ Reranker AlBERT-xlarge model, TriviaQA-trained\",\n \"skip_if_exists_path\": \"models/rerankers/reranker_tqa_xlarge\"\n },\n \"models.rerankers.reranker_tqa_xxlarge\": {\n 's3_url': 'https://dl.fbaipublicfiles.com/paq/v1/models/rerankers/reranker_tqa_xxlarge.tar.gz',\n \"original_ext\": \".tar.gz\",\n \"compressed\": True,\n \"desc\": \"RePAQ Reranker AlBERT-xxlarge model, TriviaQA-trained\",\n \"skip_if_exists_path\": \"models/rerankers/reranker_tqa_xxlarge\"\n },\n \"models.rerankers.reranker_nq_xlarge\": {\n 's3_url': 'https://dl.fbaipublicfiles.com/paq/v1/models/rerankers/reranker_nq_xlarge.tar.gz',\n \"original_ext\": \".tar.gz\",\n \"compressed\": True,\n \"desc\": \"RePAQ Reranker AlBERT-xlarge model, NQ-trained\",\n \"skip_if_exists_path\": \"models/rerankers/reranker_nq_xlarge\"\n },\n \"models.rerankers.reranker_nq_xxlarge\": {\n 's3_url': 'https://dl.fbaipublicfiles.com/paq/v1/models/rerankers/reranker_nq_xxlarge.tar.gz',\n \"original_ext\": \".tar.gz\",\n \"compressed\": True,\n \"desc\": \"RePAQ Reranker AlBERT-xxlarge model, NQ-trained\",\n \"skip_if_exists_path\": \"models/rerankers/reranker_nq_xxlarge\"\n },\n\n \"models.passage_rankers.passage_ranker_base\": {\n 's3_url': 'https://dl.fbaipublicfiles.com/paq/v1/models/passage_rankers/passage_ranker_base.tar.gz',\n \"original_ext\": \".tar.gz\",\n \"compressed\": True,\n \"desc\": \"Passage Ranker model, BERT-base model, trained on NQ passages with hard negatives\",\n \"skip_if_exists_path\": \"models/passage_rankers/passage_ranker_base\"\n },\n\n\n \"models.qgen.qgen_multi_base\": {\n 's3_url': 'https://dl.fbaipublicfiles.com/paq/v1/models/qgen/qgen_multi_base.tar.gz',\n \"original_ext\": \".tar.gz\",\n \"compressed\": True,\n \"desc\": \"Question Generator model. BART-base model, multitask-trained\",\n \"skip_if_exists_path\": \"models/qgen/qgen_multi_base\"\n },\n \"models.qgen.qgen_nq_base\": {\n 's3_url': 'https://dl.fbaipublicfiles.com/paq/v1/models/qgen/qgen_nq_base.tar.gz',\n \"original_ext\": \".tar.gz\",\n \"compressed\": True,\n \"desc\": \"Question Generator model. BART-base model, NQ-trained\",\n \"skip_if_exists_path\": \"models/qgen/qgen_nq_base\"\n },\n\n\n \"models.filtering.dpr_nq_passage_retriever\": {\n 's3_url': 'https://dl.fbaipublicfiles.com/paq/v1/models/filtering/dpr_nq_passage_retriever.tar.gz',\n \"original_ext\": \".tar.gz\",\n \"compressed\": True,\n \"desc\": \"DPR Passage retriever and faiss index, from the DPR Paper, used in global filtering, NQ-trained\",\n \"skip_if_exists_path\": \"models/filtering/dpr_nq_passage_retriever\",\n\n },\n \"models.filtering.fid_reader_nq_base\": {\n 's3_url': 'https://dl.fbaipublicfiles.com/paq/v1/models/filtering/fid_reader_nq_base.tar.gz',\n \"original_ext\": \".tar.gz\",\n \"compressed\": True,\n \"desc\": \"FID-base reader, from the Fusion-in-Decoder paper, used in global and local filtering, NQ-trained\",\n \"skip_if_exists_path\": \"models/filtering/fid_reader_nq_base\",\n },\n\n \"models.answer_extractors.answer_extractor_nq_base\": {\n 's3_url': 'https://dl.fbaipublicfiles.com/paq/v1/models/answer_extractors/answer_extractor_nq_base.tar.gz',\n \"original_ext\": \".tar.gz\",\n \"compressed\": True,\n \"desc\": \"Learnt Answer Span Extractor, BERT-base, NQ-trained \",\n \"skip_if_exists_path\": \"models/answer_extractors/answer_extractor_nq_base\",\n },\n\n\n \"predictions.retriever_results.multi_xlarge_nq_test\": {\n 's3_url': 'https://dl.fbaipublicfiles.com/paq/v1/predictions/retriever_results/multi_xlarge_nq_test.jsonl.gz',\n \"original_ext\": \".jsonl\",\n \"compressed\": True,\n \"desc\": \"Learnt Answer Span Extractor, BERT-base, NQ-trained \",\n \"skip_if_exists_path\": \"predictions/retriever_results/multi_xlarge_nq_test.jsonl\",\n },\n\n\n}\n\n\ndef untar(tar_filename: str) -> List[str]:\n logger.info(\"Uncompressing %s\", tar_filename)\n tar = tarfile.open(tar_filename)\n tar.extractall(path=os.path.dirname(tar_filename))\n tar.close()\n tar_dir = tar_filename.split('.tar.gz.tmp')[0]\n return [os.path.join(tar_dir, f) for f in os.listdir(tar_dir)]\n\n\ndef unpack(gzip_file: str, out_file: str):\n logger.info(\"Uncompressing %s\", gzip_file)\n input = gzip.GzipFile(gzip_file, \"rb\")\n s = input.read()\n input.close()\n output = open(out_file, \"wb\")\n output.write(s)\n output.close()\n logger.info(\" Saved to %s\", out_file)\n\n\ndef _get_root_dir(out_dir):\n if out_dir:\n root_dir = out_dir\n else:\n # since hydra overrides the location for the 'current dir' for every run and we don't want to duplicate\n # resources multiple times, remove the current folder's volatile part\n root_dir = os.path.abspath(\"./\")\n if \"/outputs/\" in root_dir:\n root_dir = root_dir[: root_dir.index(\"/outputs/\")]\n return root_dir\n\ndef download_resource(\n s3_url: str, original_ext: str, compressed: bool, resource_key: str, out_dir: str, use_url_fname=False,\n) -> Tuple[str, str]:\n logger.info(\"Requested resource from %s\", s3_url)\n path_names = resource_key.split(\".\")\n\n root_dir = _get_root_dir(out_dir)\n logger.info(\"Download root_dir %s\", root_dir)\n save_root = os.path.join(root_dir, \"data\", *path_names[:-1]) # last segment is for file name\n\n pathlib.Path(save_root).mkdir(parents=True, exist_ok=True)\n if use_url_fname:\n local_file_uncompressed = os.path.abspath(\n os.path.join(save_root, s3_url.split('/')[-1])\n )\n else:\n local_file_uncompressed = os.path.abspath(\n os.path.join(save_root, path_names[-1] + original_ext)\n )\n logger.info(\"File to be downloaded as %s\", local_file_uncompressed)\n\n if os.path.exists(local_file_uncompressed):\n logger.info(\"File already exist %s\", local_file_uncompressed)\n return save_root, local_file_uncompressed\n\n local_file = local_file_uncompressed if not compressed else local_file_uncompressed + '.tmp'\n wget.download(s3_url, out=local_file)\n\n logger.info(\"Downloaded to %s\", local_file)\n\n if compressed:\n if original_ext == '.tar.gz':\n local_files = untar(local_file)\n os.remove(local_file)\n local_file = ','.join(local_files)\n else:\n uncompressed_file = os.path.join(save_root, path_names[-1] + original_ext)\n unpack(local_file, uncompressed_file)\n os.remove(local_file)\n local_file = uncompressed_file\n return save_root, local_file\n\n\ndef download_file(s3_url: str, out_dir: str, file_name: str):\n logger.info(\"Loading from %s\", s3_url)\n local_file = os.path.join(out_dir, file_name)\n\n if os.path.exists(local_file):\n logger.info(\"File already exist %s\", local_file)\n return\n\n wget.download(s3_url, out=local_file)\n logger.info(\"Downloaded to %s\", local_file)\n\n\ndef download(resource_key: str, out_dir: str = None):\n if resource_key not in RESOURCES_MAP:\n # match by prefix\n resources = [k for k in RESOURCES_MAP.keys() if k.startswith(resource_key)]\n if resources:\n for key in resources:\n download(key, out_dir)\n else:\n logger.info(\"no resources found for specified key\")\n return []\n download_info = RESOURCES_MAP[resource_key]\n\n if \"skip_if_exists_path\" in download_info:\n root_dir = _get_root_dir(out_dir)\n save_root = os.path.join(root_dir, \"data\", download_info['skip_if_exists_path'])\n if os.path.exists(save_root):\n logger.info(f\"Resource: {resource_key} already exists here: {save_root}, \"\n f\"delete this directory to force re-download\")\n return []\n\n\n s3_url = download_info[\"s3_url\"]\n\n save_root_dir = None\n data_files = []\n if isinstance(s3_url, list):\n if isinstance(download_info[\"original_ext\"], str):\n exts = [download_info[\"original_ext\"] for _ in s3_url]\n else:\n exts = download_info['original_ext']\n for i, (url, ext) in enumerate(zip(s3_url, exts)):\n save_root_dir, local_file = download_resource(\n url,\n ext,\n download_info[\"compressed\"],\n resource_key,\n # \"{}_{}\".format(resource_key, i),\n out_dir,\n True\n )\n data_files.append(local_file)\n else:\n save_root_dir, local_file = download_resource(\n s3_url,\n download_info[\"original_ext\"],\n download_info[\"compressed\"],\n resource_key,\n out_dir,\n )\n data_files.append(local_file)\n\n license_files = download_info.get(\"license_files\", None)\n if license_files:\n download_file(license_files[0], save_root_dir, \"LICENSE\")\n download_file(license_files[1], save_root_dir, \"README\")\n return data_files\n\n\ndef main():\n NL = '\\n'\n parser = argparse.ArgumentParser(\"Tool for downloading resources\",formatter_class=argparse.RawTextHelpFormatter)\n\n parser.add_argument(\n \"--output_dir\",\n default=\"./\",\n type=str,\n help=\"The output directory to download file\",\n )\n parser.add_argument(\n \"--name\", \"-n\",\n type=str,\n required=True,\n help=f\"Resource name. Choose between: {NL + NL.join([str(k) + ' : ' + str(v['desc']) for k, v in RESOURCES_MAP.items()])}\",\n )\n parser.add_argument('-v', '--verbose', action=\"store_true\")\n args = parser.parse_args()\n\n if args.verbose:\n logging.basicConfig(level=logging.DEBUG)\n\n if args.name:\n downloaded_files = download(args.name, args.output_dir)\n logger.info(f'\\nDownloaded the following files for resource {args.name} :')\n for d in downloaded_files:\n if ',' in d:\n for d2 in d.split(','):\n logger.info(d2)\n else:\n logger.info(f'Downloaded {d}')\n else:\n logger.error(\"Please specify resource value. Possible options are:\")\n for k, v in RESOURCES_MAP.items():\n logger.error(\"Resource key=%s : %s\", k, v[\"desc\"])\n\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": "paq/evaluation/__init__.py", "content": "#!/usr/bin/env python3\n# Copyright (c) Facebook, Inc. and its affiliates.\n# All rights reserved.\n#\n# This source code is licensed under the license found in the\n# LICENSE file in the root directory of this source tree." }, { "path": "paq/evaluation/eval_reranker.py", "content": "#!/usr/bin/env python3\n# Copyright (c) Facebook, Inc. and its affiliates.\n# All rights reserved.\n#\n# This source code is licensed under the license found in the\n# LICENSE file in the root directory of this source tree.\nimport argparse\nfrom paq.evaluation.eval_utils import metric_max_over_ground_truths, exact_match_score\nfrom paq.paq_utils import load_jsonl\n\n\ndef evaluate_exact_match(preds, refs):\n assert len(refs) == len(preds)\n\n scores = []\n for ref, pred in zip(refs, preds):\n score = metric_max_over_ground_truths(exact_match_score, pred, ref)\n scores.append(score)\n\n return sum(scores) / len(scores)\n\n\nif __name__ == '__main__':\n parser = argparse.ArgumentParser()\n parser.add_argument('--predictions', type=str, help=\"path to predicted answers in jsonl format {'question': question, 'prediciton': predicted answer}\")\n parser.add_argument('--references', type=str, help=\"path to gold answers, in jsonl format\")\n args = parser.parse_args()\n\n refs = load_jsonl(args.references)\n preds = load_jsonl(args.predictions)\n assert len(refs) == len(preds), \"number of references doesnt match number of predictions\"\n\n assert len(refs) == len(preds)\n scores = []\n for r, p in zip(refs, preds):\n ref_answers = r['answer']\n pred_answer = p['prediction']\n score = metric_max_over_ground_truths(exact_match_score, pred_answer, ref_answers)\n scores.append(score)\n\n print(f'{100 * sum(scores) / len(scores):0.1f}% \\n({sum(scores)} / {len(scores)})')\n" }, { "path": "paq/evaluation/eval_retriever.py", "content": "#!/usr/bin/env python3\n# Copyright (c) Facebook, Inc. and its affiliates.\n# All rights reserved.\n#\n# This source code is licensed under the license found in the\n# LICENSE file in the root directory of this source tree.\nimport argparse\nfrom paq.evaluation.eval_utils import metric_max_over_ground_truths, exact_match_score\nfrom paq.paq_utils import load_jsonl\n\n\ndef eval_retriever(refs, preds, hits_at_k):\n for k in hits_at_k:\n scores = []\n dont_print = False\n for r, p in zip(refs, preds):\n if hits_at_k[-1] > len(p['retrieved_qas']):\n print(f'Skipping hits@{K} eval as {K} is larger than number of retrieved results')\n dont_print = True\n ref_answers = r['answer']\n em = any([\n metric_max_over_ground_truths(exact_match_score, pred_answer['answer'][0], ref_answers)\n for pred_answer in p['retrieved_qas'][:k]\n ])\n scores.append(em)\n\n if not dont_print:\n print(f'{k}: {100 * sum(scores) / len(scores):0.1f}% \\n({sum(scores)} / {len(scores)})')\n\n\nif __name__ == '__main__':\n parser = argparse.ArgumentParser()\n parser.add_argument('--predictions', type=str, help=\"path to retrieval results to eval, in PAQ's retrieved results jsonl format\")\n parser.add_argument('--references', type=str, help=\"path to gold answers, in jsonl format\")\n parser.add_argument('--hits_at_k', type=str, help='comma separated list of K to eval hits@k for', default=\"1,10,50\")\n args = parser.parse_args()\n\n refs = load_jsonl(args.references)\n preds = load_jsonl(args.predictions)\n assert len(refs) == len(preds), \"number of references doesnt match number of predictions\"\n\n hits_at_k = sorted([int(k) for k in args.hits_at_k.split(',')])\n eval_retriever(refs, preds, hits_at_k)\n" }, { "path": "paq/evaluation/eval_utils.py", "content": "#!/usr/bin/env python3\n# Copyright (c) Facebook, Inc. and its affiliates.\n# All rights reserved.\n#\n# This source code is licensed under the license found in the\n# LICENSE file in the root directory of this source tree.\nimport re\nimport string\nfrom typing import List, Union\n\n\ndef normalize_answer(s):\n \"\"\"Lower text and remove punctuation, articles and extra whitespace.\"\"\"\n\n def remove_articles(text):\n return re.sub(r'\\b(a|an|the)\\b', ' ', text)\n\n def white_space_fix(text):\n return ' '.join(text.split())\n\n def remove_punc(text):\n exclude = set(string.punctuation)\n return ''.join(ch for ch in text if ch not in exclude)\n\n def lower(text):\n return text.lower()\n\n return white_space_fix(remove_articles(remove_punc(lower(s))))\n\n\ndef exact_match_score(prediction, ground_truth):\n return (normalize_answer(prediction) == normalize_answer(ground_truth))\n\n\ndef metric_max_over_ground_truths(metric_fn, predictions: Union[str, List[str]], ground_truths: List[str]):\n scores_for_ground_truths = []\n\n if isinstance(predictions, str):\n predictions = [predictions]\n\n for prediction in predictions:\n for ground_truth in ground_truths:\n score = metric_fn(prediction, ground_truth)\n scores_for_ground_truths.append(score)\n\n return max(scores_for_ground_truths)\n\n" }, { "path": "paq/generation/__init__.py", "content": "#!/usr/bin/env python3\n# Copyright (c) Facebook, Inc. and its affiliates.\n# All rights reserved.\n#\n# This source code is licensed under the license found in the\n# LICENSE file in the root directory of this source tree." }, { "path": "paq/generation/answer_extractor/__init__.py", "content": "#!/usr/bin/env python3\n# Copyright (c) Facebook, Inc. and its affiliates.\n# All rights reserved.\n#\n# This source code is licensed under the license found in the\n# LICENSE file in the root directory of this source tree." }, { "path": "paq/generation/answer_extractor/extract_answers.py", "content": "#!/usr/bin/env python3\n# Copyright (c) Facebook, Inc. and its affiliates.\n# All rights reserved.\n#\n# This source code is licensed under the license found in the\n# LICENSE file in the root directory of this source tree.\nimport json\nfrom paq.paq_utils import load_jsonl, dump_jsonl, load_dpr_tsv\nfrom paq.generation.answer_extractor.extractors import load_answer_extractor\nimport logging\nimport argparse\n\nlogger = logging.getLogger(__name__)\n\n\ndef load_passages(path):\n try:\n return load_jsonl(path)\n except:\n return load_dpr_tsv(path)\n\n\ndef extract_answers(config, input_file, verbose):\n answer_extractor = load_answer_extractor(config)\n passages = load_passages(input_file)\n logger.info(\"Running answer extractor...\")\n annotations = answer_extractor.extract_answers_from_passages(passages, disable_tqdm=not verbose)\n return annotations\n\n\ndef extract_answers_and_write_to_file(config, input_path, output_path, verbose):\n annotations = extract_answers(config, input_path, verbose)\n logger.info('writing extracted answers to file...')\n dump_jsonl(annotations, output_path)\n\n\nif __name__ == '__main__':\n parser = argparse.ArgumentParser(\"Extract answers from passages\")\n parser.add_argument('--passages_to_extract_from', type=str, required=True, help='path to passages to extract in jsonl format')\n parser.add_argument('--output_path', type=str, required=True, help='Path to dump results to')\n parser.add_argument('--path_to_config', type=str, required=True, help='path to answer extractor config file')\n parser.add_argument('-v', '--verbose', action=\"store_true\")\n args = parser.parse_args()\n\n if args.verbose:\n logging.basicConfig(level=logging.DEBUG)\n\n with open(args.path_to_config) as f:\n config = json.load(f)\n\n if 'answer_extractor' in config:\n config = config['answer_extractor']\n\n extract_answers_and_write_to_file(config, args.passages_to_extract_from, args.output_path, args.verbose)\n" }, { "path": "paq/generation/answer_extractor/extractors.py", "content": "#!/usr/bin/env python3\n# Copyright (c) Facebook, Inc. and its affiliates.\n# All rights reserved.\n#\n# This source code is licensed under the license found in the\n# LICENSE file in the root directory of this source tree.\nimport logging\nimport numpy as np\nfrom typing import List, Dict\nimport torch\nfrom tqdm.auto import tqdm\nfrom transformers import AutoConfig, AutoTokenizer\nfrom paq.paq_utils import is_spacy_available\nfrom paq.generation.answer_extractor.span2D_model import AnswerSpanExtractor2DModel, postprocess_span2d_output\n\n\ndef get_output_format(all_passages, all_answers):\n all_results = []\n assert len(all_passages) == len(all_answers)\n for passage, answers in zip(all_passages, all_answers):\n result = {\n \"passage_id\": passage[\"passage_id\"],\n \"passage\": passage[\"passage\"],\n \"answers\": answers,\n \"metadata\": passage[\"metadata\"],\n }\n all_results.append(result)\n return all_results\n\n\nclass SpacyNERExtractor:\n \"\"\"\n Spacy NER extractor\n \"\"\"\n name = \"answer_extractor/spacy_ner\"\n\n def __init__(self, model=\"en_core_web_sm\"):\n assert is_spacy_available(), \"Spacy is not installed. Please install with `pip install spacy`.\"\n import spacy\n self.nlp = spacy.load(model)\n\n def extract_from_passage(self, passage: str) -> List[Dict]:\n doc = self.nlp(passage)\n answers = []\n for ent in doc.ents:\n answers.append({\n \"text\": ent.text,\n \"start\": ent.start_char,\n \"end\": ent.end_char,\n \"score\": None\n })\n return answers\n\n def extract_answers_from_passages(self, passages_to_label, disable_tqdm=False):\n all_answers = []\n for doc in tqdm(self.nlp.pipe([p['passage'] for p in passages_to_label], batch_size=128), disable=disable_tqdm):\n answers = []\n for ent in doc.ents:\n answers.append({\n \"text\": ent.text,\n \"start\": ent.start_char,\n \"end\": ent.end_char,\n \"score\": None\n })\n all_answers.append(answers)\n\n # Post-process\n all_results = get_output_format(passages_to_label, all_answers)\n return all_results\n\n\nclass Span2DAnswerExtractor:\n \"\"\"\n Predict answer spans with their joint span probability P(start, end|context).\n \"\"\"\n name = \"answer_extractor/span2D\"\n\n def __init__(\n self,\n model_path: str,\n config_path: str = None,\n tokenizer_path: str = None,\n topk: int = 5,\n max_answer_len: int = 30,\n max_seq_len: int = 256,\n doc_stride: int = 128,\n batch_size: int = 10,\n device: int = 0,\n **kwargs\n ):\n assert model_path is not None\n self.device = torch.device(f\"cuda:{device}\") if device is not None else torch.device(\"cpu\")\n\n config = AutoConfig.from_pretrained(config_path if config_path is not None else model_path)\n self.tokenizer = AutoTokenizer.from_pretrained(tokenizer_path if tokenizer_path is not None else model_path)\n self.model = AnswerSpanExtractor2DModel.from_pretrained(model_path, config=config)\n\n self.model.to(self.device)\n self.model.eval()\n\n self.topk = topk\n self.max_answer_len = max_answer_len\n self.max_seq_len = max_seq_len\n self.doc_stride = doc_stride\n logging.info(f\"Extract top {self.topk} answer spans with \"\n f\"max_answer_len={self.max_answer_len}, max_seq_len={self.max_seq_len}, \"\n f\"doc_stride={self.doc_stride}.\")\n\n self.kwargs = kwargs\n self.batch_size = batch_size\n\n def _tokenize(self, passage: str):\n input_features = self.tokenizer(\n passage,\n truncation=True,\n max_length=self.max_seq_len,\n stride=self.doc_stride,\n return_overflowing_tokens = True,\n return_offsets_mapping = True,\n padding=\"max_length\",\n )\n input_features[\"input_ids\"] = torch.tensor(input_features[\"input_ids\"]).to(self.device)\n input_features[\"token_type_ids\"] = torch.tensor(input_features[\"token_type_ids\"]).to(self.device)\n input_features[\"attention_mask\"] = torch.tensor(input_features[\"attention_mask\"]).to(self.device)\n return input_features\n\n def extract_from_passage(self, passage: str):\n input_features = self._tokenize(passage)\n model_output = self.model(**input_features, return_dict=True)\n answers = postprocess_span2d_output(model_output, input_features, self.max_answer_len, passage, self.topk)\n for answer in answers:\n answer['score'] = np.log(answer['score'])\n return answers\n\n def extract_answers_from_passages(self, passages_to_label, disable_tqdm=False):\n\n # Run the pipeline (model) to extract the answer spans\n all_answers = []\n for passage in tqdm(passages_to_label, disable=disable_tqdm):\n answers = self.extract_from_passage(passage[\"passage\"])\n all_answers.append(answers)\n\n # Post-process\n all_results = get_output_format(passages_to_label, all_answers)\n return all_results\n\n\ndef load_answer_extractor(config):\n ANS_EXT_MAP = {m.name: m for m in [SpacyNERExtractor, Span2DAnswerExtractor]}\n answer_extractor = ANS_EXT_MAP[config['name']](**config['config'])\n return answer_extractor\n" }, { "path": "paq/generation/answer_extractor/span2D_model.py", "content": "#!/usr/bin/env python3\n# Copyright (c) Facebook, Inc. and its affiliates.\n# All rights reserved.\n#\n# This source code is licensed under the license found in the\n# LICENSE file in the root directory of this source tree.\nfrom dataclasses import dataclass\nfrom typing import List, Optional, Tuple, Dict\nimport numpy as np\nimport math\n\nimport torch\nfrom torch import nn\nfrom torch.nn import BCEWithLogitsLoss, ModuleList\n\nfrom transformers import BertPreTrainedModel, BertModel\nfrom transformers.file_utils import ModelOutput\n\n\n@dataclass\nclass AnswerSpanExtractor2DModelOutput(ModelOutput):\n \"\"\"\n Base class for outputs of question answering models.\n\n Args:\n loss (:obj:`torch.FloatTensor` of shape :obj:`(1,)`, `optional`, returned when :obj:`labels` is provided):\n Total span extraction loss is the sum of a Cross-Entropy for the start and end positions.\n span_logits (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, sequence_length)`):\n Span scores (before SoftMax).\n hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``output_hidden_states=True`` is passed or when ``config.output_hidden_states=True``):\n Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer)\n of shape :obj:`(batch_size, sequence_length, hidden_size)`.\n\n Hidden-states of the model at the output of each layer plus the initial embedding outputs.\n attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``output_attentions=True`` is passed or when ``config.output_attentions=True``):\n Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads,\n sequence_length, sequence_length)`.\n\n Attentions weights after the attention softmax, used to compute the weighted average in the self-attention\n heads.\n \"\"\"\n\n loss: Optional[torch.FloatTensor] = None\n span_logits: torch.FloatTensor = None\n span_masks: torch.Tensor = None\n hidden_states: Optional[Tuple[torch.FloatTensor]] = None\n attentions: Optional[Tuple[torch.FloatTensor]] = None\n\n\nclass AnswerSpanExtractor2DModel(BertPreTrainedModel):\n _keys_to_ignore_on_load_unexpected = [r\"pooler\"]\n\n def __init__(self, config):\n super().__init__(config)\n\n self.bert = BertModel(config)\n\n # Linear mapping for start and end representation\n self.start_outputs = nn.Linear(config.hidden_size, config.span_output_size)\n self.end_outputs = nn.Linear(config.hidden_size, config.span_output_size)\n prev_out_size = config.span_output_size * 2\n\n # Add final MLP output layers to produce probabilities\n self.output_mlp = None\n mlp_sizes = getattr(config, \"output_mlp_sizes\", None)\n if mlp_sizes and len(mlp_sizes) > 0:\n self.output_mlp = ModuleList()\n for cur_size in mlp_sizes:\n self.output_mlp.append(nn.Linear(prev_out_size, cur_size))\n self.output_mlp.append(nn.ReLU())\n prev_out_size = cur_size\n\n self.span_outputs = nn.Linear(prev_out_size, 1)\n\n self.max_answer_length = getattr(config, \"max_answer_length\", 30)\n self.init_weights()\n\n def forward(\n self,\n input_ids=None,\n attention_mask=None,\n token_type_ids=None,\n position_ids=None,\n head_mask=None,\n inputs_embeds=None,\n start_positions=None,\n end_positions=None,\n output_attentions=None,\n output_hidden_states=None,\n return_dict=None,\n **kwargs,\n ):\n r\"\"\"\n start_positions (:obj:`torch.LongTensor` of shape :obj:`(batch_size, max_num_answers)`, `optional`):\n Labels for position (index) of the start of the labelled span for computing the token classification loss.\n Positions are clamped to the length of the sequence (:obj:`sequence_length`). Position outside of the\n sequence are not taken into account for computing the loss.\n end_positions (:obj:`torch.LongTensor` of shape :obj:`(batch_size, max_num_answers)`, `optional`):\n Labels for position (index) of the end of the labelled span for computing the token classification loss.\n Positions are clamped to the length of the sequence (:obj:`sequence_length`). Position outside of the\n sequence are not taken into account for computing the loss.\n \"\"\"\n return_dict = return_dict if return_dict is not None else self.config.use_return_dict\n\n outputs = self.bert(\n input_ids,\n attention_mask=attention_mask,\n token_type_ids=token_type_ids,\n position_ids=position_ids,\n head_mask=head_mask,\n inputs_embeds=inputs_embeds,\n output_attentions=output_attentions,\n output_hidden_states=output_hidden_states,\n return_dict=return_dict,\n )\n\n sequence_output = outputs[0]\n\n start_hidden = self.start_outputs(sequence_output) # [B, L, D]\n end_hidden = self.end_outputs(sequence_output) # [B, L, D]\n\n sequence_length = sequence_output.shape[1]\n start_hidden = start_hidden.unsqueeze(2).expand(-1, -1, sequence_length, -1) # [B, L, L, D]\n end_hidden = end_hidden.unsqueeze(1).expand(-1, sequence_length, -1, -1) # [B, L, L, D]\n # Concat the start and end representation to form span representation\n span_hidden = torch.cat((start_hidden, end_hidden), -1) # [B, L, L, D*2]\n\n # Run MLP layers\n if self.output_mlp is not None:\n for layer in self.output_mlp:\n span_hidden = layer(span_hidden) # [B, L, L, ?]\n\n span_logits = self.span_outputs(span_hidden) # [B, L, L, 1]\n span_logits = span_logits.squeeze(-1) # [B, L, L]\n\n span_masks = torch.einsum('bi,bj->bij', attention_mask, attention_mask) # [B, L, L]\n span_masks = torch.triu(span_masks)\n span_masks = torch.tril(span_masks, diagonal=self.max_answer_length)\n\n def _convert_to_span_matrix(start_positions, end_positions):\n span_labels = torch.zeros_like(span_logits) # [B, L, L]\n for i, (start_post, end_post) in enumerate(zip(start_positions, end_positions)):\n for start_idx, end_idx in zip(start_post, end_post):\n if 0 <= start_idx and 0 <= end_idx: # we use -1 as null indicator\n assert start_idx < sequence_length and end_idx < sequence_length\n span_labels[i, start_idx, end_idx] = 1.\n else:\n break\n return span_labels\n\n total_loss = None\n if start_positions is not None and end_positions is not None:\n span_labels = _convert_to_span_matrix(start_positions, end_positions)\n\n loss_fct = BCEWithLogitsLoss(weight=span_masks, reduction=\"sum\")\n total_loss = loss_fct(span_logits, span_labels) # / torch.sum(span_masks)\n\n if not return_dict:\n output = (span_logits,) + outputs[2:]\n return ((total_loss,) + output) if total_loss is not None else output\n\n return AnswerSpanExtractor2DModelOutput(\n loss=total_loss,\n span_logits=span_logits,\n span_masks=span_masks,\n hidden_states=outputs.hidden_states,\n attentions=outputs.attentions,\n )\n\n\ndef sigmoid(x):\n return 1 / (1 + math.exp(-x))\n\n\ndef postprocess_span2d_output(span2D_output: AnswerSpanExtractor2DModelOutput, features,\n max_answer_length, passage: str, n_best_size:int) -> List[Dict]:\n all_span_logits = span2D_output.span_logits.detach().cpu().numpy()\n all_span_masks = span2D_output.span_masks.detach().cpu().numpy()\n\n prelim_predictions = []\n # Looping through all the features associated to the current example.\n for feature_index in range(len(all_span_logits)):\n # We grab the predictions of the model for this feature.\n span_logits = all_span_logits[feature_index]\n span_masks = all_span_masks[feature_index]\n span_logits += -100 * (1 - span_masks) # mask the span logits\n\n # This is what will allow us to map some the positions in our logits to span of texts in the original\n # context.\n offset_mapping = features[\"offset_mapping\"][feature_index]\n # Optional `token_is_max_context`, if provided we will remove answers that do not have the maximum context\n # available in the current feature.\n token_is_max_context = None\n\n # Update minimum null prediction.\n feature_null_score = span_logits[0, 0]\n min_null_prediction = {\"offsets\": (0, 0), \"score\": feature_null_score}\n\n # Go through all possibilities for the `n_best_size` greater start and end logits.\n # start_indexes = np.argsort(start_logits)[-1: -n_best_size - 1: -1].tolist()\n # end_indexes = np.argsort(end_logits)[-1: -n_best_size - 1: -1].tolist()\n start_indexes, end_indexes = np.unravel_index(\n np.argsort(span_logits, axis=None)[-1:-n_best_size - 10:-1], # a buffer of 10 in case some are invalid\n span_logits.shape\n )\n start_indexes, end_indexes = start_indexes.tolist(), end_indexes.tolist()\n for start_index, end_index in zip(start_indexes, end_indexes):\n # Don't consider out-of-scope answers, either because the indices are out of bounds or correspond\n # to part of the input_ids that are not in the context.\n if (\n start_index >= len(offset_mapping)\n or end_index >= len(offset_mapping)\n or offset_mapping[start_index] is None\n or offset_mapping[end_index] is None\n ):\n continue\n # Don't consider answers with a length that is either < 0 or > max_answer_length.\n if end_index < start_index or end_index - start_index + 1 > max_answer_length:\n continue\n # Don't consider answer that don't have the maximum context available (if such information is\n # provided).\n if token_is_max_context is not None and not token_is_max_context.get(str(start_index), False):\n continue\n prelim_predictions.append(\n {\n \"offsets\": (offset_mapping[start_index][0], offset_mapping[end_index][1]),\n \"score\": span_logits[start_index, end_index],\n }\n )\n\n # Only keep the best `n_best_size` predictions.\n predictions = sorted(prelim_predictions, key=lambda x: x[\"score\"], reverse=True)[:n_best_size]\n\n # Use the offsets to gather the answer text in the original context.\n for pred in predictions:\n offsets = pred.pop(\"offsets\")\n pred[\"text\"] = passage[offsets[0]: offsets[1]]\n pred[\"start\"] = offsets[0]\n pred[\"end\"] = offsets[1]\n\n # In the very rare edge case we have not a single non-null prediction, we create a fake prediction to avoid\n # failure.\n if len(predictions) == 0 or (len(predictions) == 1 and predictions[0][\"text\"] == \"\"):\n predictions.insert(0, {\"text\": \"null\", \"score\": -100.0, \"start\": 0, \"end\": 0})\n\n # Include the probabilities in our predictions.\n for pred in predictions:\n score = pred.get(\"score\")\n pred[\"score\"] = sigmoid(score)\n\n return predictions\n" }, { "path": "paq/generation/filtering/__init__.py", "content": "#!/usr/bin/env python3\n# Copyright (c) Facebook, Inc. and its affiliates.\n# All rights reserved.\n#\n# This source code is licensed under the license found in the\n# LICENSE file in the root directory of this source tree." }, { "path": "paq/generation/filtering/filter_questions.py", "content": "#!/usr/bin/env python3\n# Copyright (c) Facebook, Inc. and its affiliates.\n# All rights reserved.\n#\n# This source code is licensed under the license found in the\n# LICENSE file in the root directory of this source tree.\nimport json\nfrom paq.paq_utils import load_jsonl, dump_jsonl\nfrom paq.generation.filtering.filterer import load_retriever, load_reader\nimport logging\nimport argparse\n\nlogger = logging.getLogger(__name__)\n\n\ndef retrieve_documents_for_generated_questions(config, input_file, verbose):\n retriever = load_retriever(config[\"retriever\"])\n generated_questions = load_jsonl(input_file)\n logger.info(\"Running Filterer Retriever...\")\n generated_questions_with_retrieved_docs = retriever.retrieve_documents(generated_questions)\n return generated_questions_with_retrieved_docs\n\n\ndef generate_answers_for_generated_questions_with_retrieved_docs(config, input_file, verbose):\n reader = load_reader(config[\"reader\"])\n generated_questions_with_retrieved_docs = load_jsonl(input_file)\n\n logger.info(\"Running Filterer Reader...\")\n results = reader.generate_answers(generated_questions_with_retrieved_docs)\n return results\n\n\ndef filter_generated_questions_and_write_to_file(config, input_path, output_path, verbose):\n results = retrieve_documents_for_generated_questions(config, input_path, verbose)\n retrieval_results_fi = output_path + '.retrieval_results'\n dump_jsonl(results, retrieval_results_fi)\n results = generate_answers_for_generated_questions_with_retrieved_docs(config, retrieval_results_fi, verbose)\n logger.info('Writing generated questions to file...')\n dump_jsonl(results, output_path)\n\n\nif __name__ == '__main__':\n parser = argparse.ArgumentParser(\"Extract answers from passages\")\n parser.add_argument('--generated_questions_to_filter',\n type=str,\n required=True,\n help='path to generate from (in jsonl format, produced by `answer_extractor`)')\n parser.add_argument('--output_path', type=str, required=True, help='Path to dump results to')\n parser.add_argument('--path_to_config', type=str, required=True, help='path to question generator config file')\n parser.add_argument('-v', '--verbose', action=\"store_true\")\n args = parser.parse_args()\n\n if args.verbose:\n logging.basicConfig(level=logging.DEBUG)\n\n with open(args.path_to_config) as f:\n config = json.load(f)\n\n if 'filterer' in config:\n config = config['filterer']\n\n filter_generated_questions_and_write_to_file(config, args.generated_questions_to_filter, args.output_path, args.verbose)\n" }, { "path": "paq/generation/filtering/filterer.py", "content": "#!/usr/bin/env python3\n# Copyright (c) Facebook, Inc. and its affiliates.\n# All rights reserved.\n#\n# This source code is licensed under the license found in the\n# LICENSE file in the root directory of this source tree.\nimport sys\nimport faiss\nimport os\n# hack to add FID to path:\nfile_path = os.path.realpath(__file__)\nfid_path = os.path.join(os.path.dirname(file_path), '../../../FiD')\nsys.path.append(fid_path)\nimport torch\nimport transformers\nimport numpy as np\nfrom torch.utils.data import DataLoader, SequentialSampler\nfrom paq.paq_utils import load_dpr_tsv, load_jsonl\nfrom transformers import AutoModel, AutoConfig, AutoTokenizer\nimport src.util\nimport src.data\nimport src.evaluation\nimport src.model\nimport logging\nfrom paq.retrievers.embed import embed\nfrom paq.retrievers.retrieve import mips\nimport pickle\nfrom torch import nn\nlogger = logging.getLogger(__name__)\n\n\ndef _load_corpus(path):\n if 'tsv' in path or 'csv' in path:\n docs = load_dpr_tsv(path)\n else:\n docs = load_jsonl(path)\n logger.info('Parsed Corpus for retrieval')\n return {d['passage_id']: {'title': d['metadata']['title'], 'text': d['passage']} for d in docs}\n\n\nclass DummyFilteringRetriever:\n \"\"\"Dummy filterer - does not retrieve any evidence\"\"\"\n name = \"filtering/dummy_filtering_retriever\"\n\n def retrieve_documents(self, data):\n return [{'question': d['question'], 'answers': [d['answer']], 'ctxs': [], 'metadata': d} for d in data]\n\n\nclass LocalFilteringRetriever:\n \"\"\"Retrieves a single document (the gold context the question was generated from\"\"\"\n name = \"filtering/local_filtering_retriever\"\n corpus = None\n\n def __init__(self, corpus_path):\n self.corpus_path = corpus_path\n\n def retrieve_documents(self, data):\n self.corpus = _load_corpus(self.corpus_path) if self.corpus is None else self.corpus\n\n examples = []\n for d in data:\n assert d['passage_id'] in self.corpus\n gold_doc = self.corpus[d['passage_id']]\n examples.append(\n {'question': d['question'].strip(), 'answers': [d['answer']], 'ctxs': [gold_doc], 'metadata': d}\n )\n return examples\n\n\nclass DPRQuestionEncoder(nn.Module):\n \"\"\"simple wrapper on DPR Question Encoder Bert model\"\"\"\n\n def __init__(self, model):\n super().__init__()\n self.model = model\n\n def forward(self, *args, **kwargs):\n seq_outputs = self.model(*args, **kwargs)['last_hidden_state']\n return seq_outputs[:, 0]\n\n\nclass GlobalFilteringRetriever:\n \"\"\"Uses DPR to retrieve relevant documents for the question\"\"\"\n name = \"filtering/global_filtering_retriever\"\n corpus = None\n index_id_to_db_id = None\n index = None\n\n def __init__(self,\n corpus_path,\n index_path,\n index_id_to_db_id_path,\n model_path,\n batch_size,\n n_queries_to_parallelize,\n max_seq_len,\n n_docs,\n device\n ):\n self.corpus_path = corpus_path\n self.index_path = index_path\n self.index_id_to_db_id_path = index_id_to_db_id_path\n self.n_docs = n_docs\n self.device = torch.device(f\"cuda:{device}\") if device is not None else torch.device(\"cpu\")\n\n config = AutoConfig.from_pretrained(model_path)\n self.tokenizer = AutoTokenizer.from_pretrained(model_path, config=config)\n self.model = DPRQuestionEncoder(AutoModel.from_pretrained(model_path, config=config))\n\n self.model.to(self.device)\n self.model.eval()\n self.batch_size = batch_size\n self.n_queries_to_parallelize = n_queries_to_parallelize\n self.max_seq_len = max_seq_len\n\n def _load_corpus(self):\n logger.info(\"Loading Corpus if not already loaded...\")\n self.corpus = _load_corpus(self.corpus_path) if self.corpus is None else self.corpus\n logger.info(\"Loading Faiss index if not already loaded...\")\n self.index = faiss.read_index(self.index_path) if self.index is None else self.index\n if self.index_id_to_db_id is None:\n with open(self.index_id_to_db_id_path, 'rb') as f:\n self.index_id_to_db_id = pickle.load(f)\n\n def retrieve_documents(self, qa_pairs):\n self._load_corpus()\n examples = []\n for ci in range(0, len(qa_pairs), self.n_queries_to_parallelize):\n chunk_examples = qa_pairs[ci: ci + self.n_queries_to_parallelize]\n queries = embed(self.model, self.tokenizer, chunk_examples, bsz=self.batch_size)\n top_indices, _ = mips(self.index, queries, self.n_docs, self.n_queries_to_parallelize)\n for ati, d in zip(top_indices, chunk_examples):\n ctxs = [self.corpus[self.index_id_to_db_id[ati[j]]] for j in range(self.n_docs)]\n examples.append({'question': d['question'], 'answers': [d['answer']], 'ctxs': ctxs, 'metadata': d})\n return examples\n\n\nclass CompatableEncoderWrapper(torch.nn.Module):\n \"\"\"Patched version of fid.model.EncoderWrapper to make it compatable with our version of transformers\"\"\"\n\n def __init__(self, encoder, use_checkpoint=False):\n super().__init__()\n self.encoder = encoder\n\n def forward(self, input_ids=None, attention_mask=None, **kwargs, ):\n # total_length = n_passages * passage_length\n bsz, total_length = input_ids.shape\n passage_length = total_length // self.n_passages\n input_ids = input_ids.view(bsz * self.n_passages, passage_length)\n attention_mask = attention_mask.view(bsz * self.n_passages, passage_length)\n outputs = self.encoder(input_ids, attention_mask, **kwargs)\n outputs.last_hidden_state = outputs.last_hidden_state.view(bsz, self.n_passages * passage_length, -1)\n return outputs\n\n\nclass FIDReader:\n \"\"\"FID Filterer\"\"\"\n name = \"filtering/fid_reader\"\n\n def __init__(self,\n model_path: str,\n batch_size: int = 10,\n device: int = 0,\n max_seq_len: int = 200,\n n_docs:int = 50,\n ):\n self.device = torch.device(f\"cuda:{device}\") if device is not None else torch.device(\"cpu\")\n self.tokenizer = transformers.T5Tokenizer.from_pretrained('t5-base', return_dict=False)\n self.model = src.model.FiDT5.from_pretrained(model_path)\n\n self.model.to(self.device)\n self.model.eval()\n self.model.encoder = CompatableEncoderWrapper(self.model.encoder.encoder) # hack to make FID compatable with newer transformers version\n self.batch_size = batch_size\n self.max_seq_len = max_seq_len\n self.n_docs = n_docs\n self.collator = src.data.Collator(self.max_seq_len, self.tokenizer)\n\n def _get_dataloader_for_examples(self, examples):\n for k, example in enumerate(examples):\n example['id'] = k\n for c in example['ctxs']:\n c['score'] = 1.0 / (k + 1)\n\n eval_dataset = src.data.Dataset(examples, self.n_docs)\n\n eval_sampler = SequentialSampler(eval_dataset)\n eval_dataloader = DataLoader(\n eval_dataset,\n sampler=eval_sampler,\n batch_size=self.batch_size,\n num_workers=20,\n collate_fn=self.collator\n )\n return eval_dataset, eval_dataloader\n\n def generate_answers(self, examples):\n\n eval_dataset, eval_dataloader = self._get_dataloader_for_examples(examples)\n total = 0\n exactmatch = []\n with torch.no_grad():\n for i, batch in enumerate(eval_dataloader):\n (idx, _, _, context_ids, context_mask) = batch\n\n outputs = self.model.generate(\n input_ids=context_ids.to(self.device),\n attention_mask=context_mask.to(self.device),\n max_length=10,\n )\n for k, o in enumerate(outputs):\n ans = self.tokenizer.decode(o, skip_special_tokens=True)\n example = eval_dataset.data[idx[k]]\n score = src.evaluation.ems(ans, example['answers'])\n exactmatch.append(score)\n example['consistent'] = score\n example['filter_answer'] = ans\n total += 1\n\n if (i + 1) % 10 == 0:\n logger.info(f'FID filtering: {i+1} / {len(eval_dataloader)} | ave = {np.mean(exactmatch):.3f}')\n logger.info(f'FID filtering: {i+1} / {len(eval_dataloader)} | ave = {np.mean(exactmatch):.3f}')\n output = _get_reader_output_format(examples)\n return output\n\n\nclass DummyReader:\n \"\"\"Dummy Reader, always returns consistent\"\"\"\n name = \"filtering/dummy_reader\"\n\n def generate_answers(self, examples):\n for example in examples:\n example['consistent'] = True\n example['filter_answer'] = \"DUMMY_READER_ANSWER\"\n output = _get_reader_output_format(examples)\n return output\n\n\ndef _get_reader_output_format(dataset):\n out = []\n for e in dataset:\n o = e['metadata']\n o['metadata'] = o.get('metadata', {})\n o['metadata']['consistent'] = e['consistent']\n o['metadata']['filter_answer'] = e['filter_answer']\n out.append(o)\n return out\n\n\ndef load_reader(config):\n READER_MAP = {m.name: m for m in [DummyReader, FIDReader]}\n reader = READER_MAP[config['name']](**config['config'])\n return reader\n\n\ndef load_retriever(config):\n RETRIEVER_MAP = {m.name: m for m in [LocalFilteringRetriever, GlobalFilteringRetriever, DummyFilteringRetriever]}\n retriever = RETRIEVER_MAP[config['name']](**config['config'])\n return retriever\n\n" }, { "path": "paq/generation/generate_qa_pairs.py", "content": "#!/usr/bin/env python3\n# Copyright (c) Facebook, Inc. and its affiliates.\n# All rights reserved.\n#\n# This source code is licensed under the license found in the\n# LICENSE file in the root directory of this source tree.\nimport json\nimport os\nfrom collections import defaultdict\nimport logging\nimport argparse\n\nfrom paq.paq_utils import load_jsonl, dump_jsonl, get_submitit_executor\nfrom paq.generation.passage_scorer.score_passages import score_passages_and_write_to_file\nfrom paq.generation.answer_extractor.extract_answers import extract_answers_and_write_to_file\nfrom paq.generation.question_generator.generate_questions import generate_questions_and_write_to_file\nfrom paq.generation.filtering.filter_questions import filter_generated_questions_and_write_to_file\n\n\nlogger = logging.getLogger(__name__)\n\nCONFIG_FILE = \"config.json\"\n\nFINAL_OUTPUT = \"final_qas.jsonl\"\nFINAL_DONE = \"FINAL_DONE\"\n\n\ndef touch(path):\n \"\"\"Create an empty file. Update the mtime if it exists.\"\"\"\n with open(path, 'a'):\n os.utime(path, None)\n\n\ndef _run_pipeline_step(config, input_file, output_file, done_indicator, verbose, fun):\n if not os.path.exists(done_indicator):\n fun(config, input_file, output_file, verbose)\n touch(done_indicator)\n return output_file\n\n\ndef run_passage_scoring(config, input_file, output_dir, verbose=False):\n output_file = os.path.join(output_dir, \"ps.jsonl\")\n done_path = os.path.join(output_dir, \"PS_DONE\")\n func = score_passages_and_write_to_file\n return _run_pipeline_step(config['passage_scorer'], input_file, output_file, done_path, verbose, func)\n\n\ndef run_answer_extraction(config, input_file, output_dir, verbose=False):\n output_file = os.path.join(output_dir, \"ae.jsonl\")\n done_path = os.path.join(output_dir, \"AE_DONE\")\n func = extract_answers_and_write_to_file\n return _run_pipeline_step(config['answer_extractor'], input_file, output_file, done_path, verbose, func)\n\n\ndef run_question_generation(config, input_file, output_dir, verbose=False):\n output_file = os.path.join(output_dir, \"qg.jsonl\")\n done_path = os.path.join(output_dir, \"QG_DONE\")\n func = generate_questions_and_write_to_file\n return _run_pipeline_step(config['question_generator'], input_file, output_file, done_path, verbose, func)\n\n\ndef run_filtering(config, input_file, output_dir, verbose=False):\n output_file = os.path.join(output_dir, \"filterd_qg.jsonl\")\n done_path = os.path.join(output_dir, \"FILTERED_DONE\")\n func = filter_generated_questions_and_write_to_file\n return _run_pipeline_step(config['filterer'], input_file, output_file, done_path, verbose, func)\n\n\ndef combine_generated_files(document_ranker_file,\n question_generation_file,\n output_file\n ):\n # Write final generated QA-pairs to an output file\n\n def _get_passage_score_map(doc_ranker_file):\n passage_scores = {}\n with open(doc_ranker_file, \"r\") as f:\n for line in f.readlines():\n row = json.loads(line)\n passage_scores[row[\"passage_id\"]] = row[\"metadata\"].get(\"ps_score\", None)\n return passage_scores\n\n def _add_passage_metadata(questions_fi, passage_scores):\n generated_qas = load_jsonl(questions_fi)\n qas_dict = defaultdict(list)\n for qas in generated_qas:\n question, answer, passage_id = qas[\"question\"], qas[\"answer\"], qas[\"passage_id\"]\n metadata = {\"passage_id\": passage_id, \"ps_score\": passage_scores[passage_id], 'answer': answer}\n metadata.update(qas[\"metadata\"])\n qas_dict[question].append((answer, metadata))\n return qas_dict\n\n def _get_output_format(qas_dict):\n final_qas = []\n for question, answers_meta in qas_dict.items():\n answers, metadata_list = zip(*answers_meta)\n final_qa = {\"question\": question, \"answer\": answers, \"metadata\": metadata_list}\n final_qas.append(final_qa)\n return final_qas\n\n passage_score_map = _get_passage_score_map(document_ranker_file)\n qas_with_meta = _add_passage_metadata(question_generation_file, passage_score_map)\n final_qas = _get_output_format(qas_with_meta)\n dump_jsonl(final_qas, output_file)\n\n\ndef run_paq_generation_pipeline(config: dict, input_file: str, output_dir: str, verbose: bool = False):\n if not os.path.exists(output_dir):\n os.makedirs(output_dir)\n\n # Save the config\n config[\"source\"], config['output_dir'] = input_file, output_dir\n with open(os.path.join(output_dir, CONFIG_FILE), \"w\") as cf:\n json.dump(config, cf, indent=2)\n\n # Run the pipeline:\n passages_fi = run_passage_scoring(config, input_file, output_dir, verbose=verbose)\n answers_fi = run_answer_extraction(config, passages_fi, output_dir, verbose=verbose)\n questions_fi = run_question_generation(config, answers_fi, output_dir, verbose=verbose)\n filtered_questions_fi = run_filtering(config, questions_fi, output_dir, verbose=verbose)\n\n # Write final generated QA-pairs to an output file\n output_fi = os.path.join(output_dir, FINAL_OUTPUT)\n logging.info(f\"Writing generated QA pairs to {output_fi}...\")\n\n final_indicator = os.path.join(output_dir, FINAL_DONE)\n if not os.path.exists(final_indicator):\n output_fi = os.path.join(output_dir, FINAL_OUTPUT)\n combine_generated_files(passages_fi, filtered_questions_fi, output_fi)\n touch(final_indicator)\n\n\ndef _is_job_finished(job_number, output_dir):\n if os.path.exists(os.path.join(output_dir, FINAL_DONE)):\n print(f'not launching job {job_number} as its already finished: ', os.path.join(output_dir, FINAL_DONE))\n\n\nif __name__ == \"__main__\":\n parser = argparse.ArgumentParser()\n parser.add_argument(\"--path_to_config\", help=\"Path of config file\")\n parser.add_argument(\"--passage_files_to_generate\", help=\"comma separated list of files to generate QA pairs from\")\n parser.add_argument(\"--output_dirs\", help=\"comma separated list of directories to write the generated QA pairs to\")\n parser.add_argument('--n_jobs', type=int, required=True, help='how many parallel jobs to use in slurm (n_jobs=-1 will run locally)')\n parser.add_argument('--slurm_partition', type=str, default=\"learnfair\", help='If using submitit to run slurm jobs, define cluster partition here')\n parser.add_argument('--slurm_comment', type=str, default=\"\", help='If using submitit to run slurm jobs, define job comment heree')\n parser.add_argument('-v', '--verbose', action=\"store_true\")\n args = parser.parse_args()\n\n if args.verbose:\n logging.basicConfig(level=logging.DEBUG)\n\n with open(args.path_to_config) as f:\n config = json.load(f)\n\n input_files = args.passage_files_to_generate.split(',')\n output_dirs = args.output_dirs.split(',')\n\n if args.n_jobs == -1:\n # Run locally\n for i, (inf, out_dir) in enumerate(zip(input_files, output_dirs)):\n if not _is_job_finished(i, out_dir):\n logging.info(f'Running generation job {i}:\\ninput file: {inf} \\nSaving results to: {out_dir}')\n run_paq_generation_pipeline(config, inf, out_dir, args.verbose)\n else:\n # Run with submitit\n executor = get_submitit_executor(n_jobs=args.n_jobs, comment=args.slurm_comment, partition=args.slurm_partition)\n jobs = []\n with executor.batch():\n for i, (inf, out_dir) in enumerate(zip(input_files, output_dirs)):\n if not _is_job_finished(i, out_dir):\n job = executor.submit(run_paq_generation_pipeline, config, inf, out_dir, args.verbose)\n jobs.append((job, inf, out_dir))\n\n logging.info('Launching the following jobs:')\n for job, inf, out_dir in jobs:\n logging.info(f'{job.job_id} {inf} -> {out_dir}')\n" }, { "path": "paq/generation/passage_scorer/__init__.py", "content": "#!/usr/bin/env python3\n# Copyright (c) Facebook, Inc. and its affiliates.\n# All rights reserved.\n#\n# This source code is licensed under the license found in the\n# LICENSE file in the root directory of this source tree." }, { "path": "paq/generation/passage_scorer/score_passages.py", "content": "#!/usr/bin/env python3\n# Copyright (c) Facebook, Inc. and its affiliates.\n# All rights reserved.\n#\n# This source code is licensed under the license found in the\n# LICENSE file in the root directory of this source tree.\nimport json\nfrom paq.paq_utils import load_jsonl, dump_jsonl, load_dpr_tsv\nfrom paq.generation.passage_scorer.scorer import load_passage_scorer\nimport logging\nimport argparse\n\nlogger = logging.getLogger(__name__)\n\n\ndef load_passages(path):\n try:\n return load_jsonl(path)\n except:\n return load_dpr_tsv(path)\n\n\ndef score_passages(config, input_file, verbose):\n passage_scorer = load_passage_scorer(config)\n passages = load_passages(input_file)\n logger.info(\"Running Passage Scorer...\")\n annotations = passage_scorer.score_passages(passages, disable_tqdm=not verbose)\n return annotations\n\n\ndef score_passages_and_write_to_file(config, input_path, output_path, verbose):\n annotations = score_passages(config, input_path, verbose)\n logger.info('writing extracted answers to file...')\n dump_jsonl(annotations, output_path)\n\n\nif __name__ == '__main__':\n parser = argparse.ArgumentParser(\"Extract answers from passages\")\n parser.add_argument('--passages_to_score', type=str, required=True, help='path to passages to extract in jsonl format')\n parser.add_argument('--output_path', type=str, required=True, help='Path to dump results to')\n parser.add_argument('--path_to_config', type=str, required=True, help='path to answer extractor config file')\n parser.add_argument('-v', '--verbose', action=\"store_true\")\n args = parser.parse_args()\n\n if args.verbose:\n logging.basicConfig(level=logging.DEBUG)\n\n with open(args.path_to_config) as f:\n config = json.load(f)\n\n if 'passage_scorer' in config:\n config = config['passage_scorer']\n\n score_passages_and_write_to_file(config, args.passages_to_score, args.output_path, args.verbose)\n" }, { "path": "paq/generation/passage_scorer/scorer.py", "content": "#!/usr/bin/env python3\n# Copyright (c) Facebook, Inc. and its affiliates.\n# All rights reserved.\n#\n# This source code is licensed under the license found in the\n# LICENSE file in the root directory of this source tree.\nfrom typing import List, Dict, Union\nfrom tqdm.auto import tqdm\nfrom transformers import AutoConfig, AutoTokenizer, AutoModelForSequenceClassification\nimport torch\n\n\nclass DummyPassageScorer:\n \"\"\"\n Dummy scorer that will always return the same score for any passage.\n \"\"\"\n name = \"passage_scorer/dummy\"\n\n def __init__(self, default_score=0.0):\n self.default_score = default_score\n\n def score_passage(self, passage: Dict) -> float:\n return self.default_score\n\n def score_passages(self, passages_to_label, disable_tqdm=False):\n for passage in tqdm(passages_to_label, disable=disable_tqdm):\n score = self.score_passage(passage)\n passage['metadata']['ps_score'] = score\n return passages_to_label\n\n\nclass LookupPassageScorer:\n \"\"\"\n Lookup scorer that will return the score from a file of precomputed passage scores for passages, or if not present, return a default score.\n \"\"\"\n name = \"passage_scorer/lookup\"\n\n def __init__(self, scores_file, default_score=-10000.0):\n self._load_passage_scores(scores_file)\n self.default_score = default_score\n\n def _load_passage_scores(self, scores_file):\n self.passage_scores = {}\n for line in open(scores_file):\n k, v = line.strip('\\n').split('\\t')\n self.passage_scores[k] = v\n\n def score_passage(self, passage: Dict) -> float:\n return self.passage_scores.get(passage['passage_id'], self.default_score)\n\n def score_passages(self, passages_to_label, disable_tqdm=False):\n for passage in tqdm(passages_to_label, disable=disable_tqdm):\n score = self.score_passage(passage)\n passage['metadata']['ps_score'] = score\n return passages_to_label\n\n\nclass LearntPassageScorer:\n \"\"\"Learnt scorer\"\"\"\n name = \"passage_scorer/learnt\"\n\n def __init__(self,\n model_path: str,\n config_path: str,\n tokenizer_path: str = None,\n batch_size: int = 10,\n device: int = 0,\n max_seq_len: int = 256):\n self.device = torch.device(f\"cuda:{device}\") if device is not None else torch.device(\"cpu\")\n\n config = AutoConfig.from_pretrained(config_path if config_path is not None else model_path)\n self.tokenizer = AutoTokenizer.from_pretrained(tokenizer_path, config=config)\n self.model = AutoModelForSequenceClassification.from_pretrained(model_path, config=config)\n\n self.model.to(self.device)\n self.model.eval()\n self.batch_size = batch_size\n self.max_seq_len = max_seq_len\n\n def _tokenize(self, texts):\n input_features = self.tokenizer.batch_encode_plus(\n texts, return_tensors='pt', padding=True, add_special_tokens=True, max_length=256, truncation=True\n )\n input_features = {k: v.to(self.device) for k, v in input_features.items()}\n return input_features\n\n def score_passages(self, passages_to_label, disable_tqdm=False):\n\n def _run_batch(batch):\n inputs = self._tokenize([b['passage'] for b in batch])\n scores = self.model(**inputs)\n log_probs = torch.log_softmax(scores.logits, dim=-1)[:, 1].cpu().tolist()\n for s, b in zip(log_probs, batch):\n b['metadata']['ps_score'] = float(s)\n return scores\n\n batch, outputs = [], []\n for passage in tqdm(passages_to_label, disable=disable_tqdm):\n batch.append(passage)\n\n if len(batch) == self.batch_size:\n _run_batch(batch)\n outputs += batch\n batch = []\n\n if len(batch) != 0:\n _run_batch(batch)\n outputs += batch\n\n return outputs\n\n\ndef load_passage_scorer(config):\n PASSAGE_SCORER_MAP = {m.name: m for m in [LearntPassageScorer, DummyPassageScorer, LookupPassageScorer]}\n answer_extractor = PASSAGE_SCORER_MAP[config['name']](**config['config'])\n return answer_extractor\n" }, { "path": "paq/generation/question_generator/__init__.py", "content": "#!/usr/bin/env python3\n# Copyright (c) Facebook, Inc. and its affiliates.\n# All rights reserved.\n#\n# This source code is licensed under the license found in the\n# LICENSE file in the root directory of this source tree." }, { "path": "paq/generation/question_generator/generate_questions.py", "content": "#!/usr/bin/env python3\n# Copyright (c) Facebook, Inc. and its affiliates.\n# All rights reserved.\n#\n# This source code is licensed under the license found in the\n# LICENSE file in the root directory of this source tree.\nimport json\nfrom paq.paq_utils import load_jsonl, dump_jsonl\nfrom paq.generation.question_generator.generator import load_question_generator\nimport logging\nimport argparse\n\nlogger = logging.getLogger(__name__)\n\n\ndef generate_questions(config, input_file, verbose):\n question_generator = load_question_generator(config)\n passage_answer_pairs = load_jsonl(input_file)\n logger.info(\"Running Question Generation...\")\n annotations = question_generator.generate_questions_from_passage_answer_pairs(passage_answer_pairs, disable_tqdm=not verbose)\n return annotations\n\n\ndef generate_questions_and_write_to_file(config, input_path, output_path, verbose):\n annotations = generate_questions(config, input_path, verbose)\n logger.info('writing generated questions to file...')\n dump_jsonl(annotations, output_path)\n\n\nif __name__ == '__main__':\n parser = argparse.ArgumentParser(\"Extract answers from passages\")\n parser.add_argument('--passage_answer_pairs_to_generate_from',\n type=str,\n required=True,\n help='path to generate from (in jsonl format, produced by `answer_extractor`)')\n parser.add_argument('--output_path', type=str, required=True, help='Path to dump results to')\n parser.add_argument('--path_to_config', type=str, required=True, help='path to question generator config file')\n parser.add_argument('-v', '--verbose', action=\"store_true\")\n args = parser.parse_args()\n\n if args.verbose:\n logging.basicConfig(level=logging.DEBUG)\n\n with open(args.path_to_config) as f:\n config = json.load(f)\n\n if 'question_generator' in config:\n config = config['question_generator']\n\n generate_questions_and_write_to_file(config, args.passage_answer_pairs_to_generate_from, args.output_path, args.verbose)\n" }, { "path": "paq/generation/question_generator/generator.py", "content": "#!/usr/bin/env python3\n# Copyright (c) Facebook, Inc. and its affiliates.\n# All rights reserved.\n#\n# This source code is licensed under the license found in the\n# LICENSE file in the root directory of this source tree.\nimport logging\nfrom typing import List, Union, Set\nfrom tqdm.auto import tqdm\nimport warnings\nimport torch\n\nwarnings.simplefilter(action='ignore', category=FutureWarning)\nwarnings.simplefilter(action='ignore', category=UserWarning)\n\nfrom transformers import AutoConfig, AutoModelForSeq2SeqLM, AutoTokenizer\nfrom transformers.pipelines import Text2TextGenerationPipeline\nfrom paq.paq_utils import to_fp16\n\n\nlogger = logging.getLogger(__name__)\n\n\ndef _batch_iterator(context_answer_pairs,\n batch_size,\n include_title: bool = True,\n ):\n\n def _answer_context_pair_2_text(answer, context):\n answer_start, answer_end, answer_text = answer[\"start\"], answer['end'], answer['text']\n return context[:answer_start] + \"** \" + context[answer_start:answer_end] + \" **\" + context[answer_end:]\n\n def _create_input_text(context, answer, title=None) -> str:\n text = _answer_context_pair_2_text(answer, context)\n\n if title is not None:\n output = f\"answer: {answer['text']} | title: {title} | context: {text}\"\n else:\n output = f\"answer: {answer['text']} | context: {text}\"\n return output\n\n iter_batch = []\n for context_answer_pair in context_answer_pairs:\n\n passage_id = context_answer_pair[\"passage_id\"]\n context = context_answer_pair[\"passage\"]\n answers = context_answer_pair[\"answers\"]\n title = context_answer_pair[\"metadata\"][\"title\"] if include_title else None\n\n for answer in answers:\n input_text = _create_input_text(context, answer, title)\n iter_batch.append((passage_id, answer, input_text))\n\n if len(iter_batch) >= batch_size:\n yield iter_batch\n iter_batch = []\n\n if len(iter_batch) > 0:\n yield iter_batch\n\n\nclass QuestionGenerator:\n name = \"question_generator/standard\"\n\n def __init__(\n self,\n model_path: str,\n config_path: str = None,\n tokenizer_path: str = None,\n include_title: bool = True,\n num_beams: int = None,\n num_return_sequences: int = 1,\n max_question_len: int = 30,\n batch_size: int = 1,\n device: int = 0,\n **kwargs\n ):\n assert model_path is not None\n\n super().__init__()\n\n config = AutoConfig.from_pretrained(config_path if config_path is not None else model_path)\n tokenizer = AutoTokenizer.from_pretrained(tokenizer_path if tokenizer_path is not None else model_path)\n model = AutoModelForSeq2SeqLM.from_pretrained(model_path, config=config)\n\n if kwargs.get('fp16', False):\n model = model.cuda()\n model = to_fp16(model)\n\n self.pipeline = Text2TextGenerationPipeline(model=model, tokenizer=tokenizer, task=\"question-generation\",\n device=device)\n\n self.include_title = include_title # include title in the source sequence\n self.num_beams = num_beams\n self.num_return_sequences = num_return_sequences\n self.max_question_len = max_question_len\n logger.info(\n f\"Generate {self.num_return_sequences} questions for each passage with beam size {self.num_beams}.\")\n\n self.batch_size = batch_size\n\n self.kwargs = kwargs\n\n def generate_question(self, data: Union[str, List[str]]):\n \"\"\"\n Generate question for a single input sequence or a batch of input sequences.\n \"\"\"\n if isinstance(data, str):\n data = [data]\n\n all_records = self.pipeline(\n data,\n return_text=True,\n # return_scores=True,\n clean_up_tokenization_spaces=True,\n max_length=self.max_question_len,\n min_length=3,\n num_beams=self.num_beams,\n num_return_sequences=self.num_return_sequences,\n **self.kwargs\n )\n\n assert len(all_records) == len(data) * self.num_return_sequences\n\n generated_questions = [r[\"generated_text\"].strip() for r in all_records]\n scores = [r.get(\"score\", None) for r in all_records]\n\n batched_questions = [\n generated_questions[i:i + self.num_return_sequences]\n for i in range(0, len(generated_questions), self.num_return_sequences)\n ]\n batched_scores = [\n scores[i:i + self.num_return_sequences]\n for i in range(0, len(scores), self.num_return_sequences)\n ]\n\n return batched_questions, batched_scores\n\n def generate_questions_from_passage_answer_pairs(self, passage_answer_pairs, disable_tqdm=False):\n outputs = []\n for batch in tqdm(\n _batch_iterator(passage_answer_pairs, self.batch_size, include_title=self.include_title),\n disable=disable_tqdm,\n total=len(passage_answer_pairs) // self.batch_size\n ):\n # try:\n batch_ids, batch_answers, batch_inputs = zip(*batch)\n batch_questions, batch_scores = self.generate_question(list(batch_inputs))\n # except Exception as e:\n # logging.info('skipping Broken batch')\n # continue\n\n for passage_id, answer, questions, scores in zip(batch_ids, batch_answers, batch_questions,\n batch_scores):\n for question, score in zip(questions, scores):\n output = {\n \"passage_id\": passage_id,\n \"answer\": answer[\"text\"],\n \"question\": question,\n \"metadata\": {\n \"answer_start\": answer[\"start\"],\n \"answer_end\": answer[\"end\"],\n \"ae_score\": answer[\"score\"],\n \"qg_score\": score,\n },\n }\n outputs.append(output)\n return outputs\n\n\ndef load_question_generator(config):\n return QuestionGenerator(**config['config'])\n" }, { "path": "paq/paq_utils.py", "content": "#!/usr/bin/env python3\n# Copyright (c) Facebook, Inc. and its affiliates.\n# All rights reserved.\n#\n# This source code is licensed under the license found in the\n# LICENSE file in the root directory of this source tree.\nimport json\nimport logging\nimport torch\nimport glob\nimport os\nimport csv\ntry:\n import submitit\n _has_submitit = True\nexcept ImportError:\n _has_submitit = False\ntry:\n import apex\n from apex import amp\n apex.amp.register_half_function(torch, \"einsum\")\n _has_apex = True\nexcept ImportError:\n _has_apex = False\ntry:\n import spacy\n from spacy.util import minibatch, compounding\n spacy.prefer_gpu()\n\n _has_spacy = True\nexcept (ImportError, AttributeError):\n _has_spacy = False\n\n\nlogger = logging.getLogger(__name__)\n\n\ndef is_spacy_available():\n return _has_spacy\n\n\ndef is_submitit_available():\n return _has_submitit\n\n\ndef is_apex_available():\n return _has_apex\n\n\ndef to_fp16(model):\n if is_apex_available():\n model = amp.initialize(model, opt_level=\"O1\")\n else:\n model = model.half()\n return model\n\n\ndef load_jsonl_memory_friendly(fi):\n logging.info(f'Loading {fi}')\n\n results = []\n for ln, line in enumerate(open(fi)):\n results.append(json.loads(line))\n logging.info(f'Loaded {ln + 1} Items from {fi}') if ln % 1000000 == 0 else None\n\n logging.info(f'Loaded {ln + 1} Items from {fi}')\n return results\n\n\ndef load_jsonl_fast(fi):\n logging.info(f'Loading {fi}')\n\n results = []\n with open(fi) as f:\n txt = f.read()\n logging.info(f'{fi} Loaded, splitting into lines...')\n lines = [t for t in txt.split('\\n') if t.strip()!='']\n logging.info(f'Parsing {len(lines)} items from jsonl:')\n\n for ln, line in enumerate(lines):\n results.append(json.loads(line))\n logging.info(f'Loaded {ln + 1} Items from {fi}') if ln % 1000000 == 0 else None\n\n logging.info(f'Loaded {ln + 1} Items from {fi}')\n return results\n\n\ndef load_jsonl(fi, memory_friendly=False):\n if memory_friendly:\n return load_jsonl_memory_friendly(fi)\n else:\n return load_jsonl_fast(fi)\n\n\ndef dump_jsonl(items, fi):\n logging.info(f'Dumping {len(items)} items into {fi}')\n k = 0\n\n with open(fi, 'w') as f:\n for k, item in enumerate(items):\n f.write(json.dumps(item) + '\\n')\n logging.info(f'Written {k + 1} / {len(items)} items') if k % 10000 == 0 else None\n\n logging.info(f'Written {k + 1} / {len(items)} items')\n\n\ndef load_dpr_tsv(fi):\n items = []\n with open(fi) as ifile:\n reader = csv.reader(ifile, delimiter='\\t')\n for spl in reader:\n idd, text, title = spl\n items.append({'passage_id': idd, \"passage\": text, \"metadata\": {'title': title}})\n return items\n\n\ndef get_vectors_file_paths_in_vector_directory(embeddings_dir):\n paths = glob.glob(os.path.abspath(embeddings_dir) + '/*')\n np = len(paths)\n template = '.'.join(paths[0].split('.')[:-1])\n return [template + f'.{j}' for j in range(np)]\n\n\ndef parse_vectors_from_directory_chunks(embeddings_dir, half):\n paths = get_vectors_file_paths_in_vector_directory(embeddings_dir)\n for j, p in enumerate(paths):\n logger.info(f'Loading vectors from {p} ({j+1} / {len(paths)})')\n m = torch.load(p)\n assert int(p.split('.')[-1]) == j, (p, j)\n\n if half:\n m = m if m.dtype == torch.float16 else m.half()\n else:\n m = m if m.dtype == torch.float32 else m.float()\n yield m\n\n\ndef parse_vectors_from_directory_fast(embeddings_dir):\n ms = []\n for m in parse_vectors_from_directory_chunks(embeddings_dir):\n ms.append(m)\n\n out = torch.cat(ms)\n logger.info(f'loaded index of shape {out.shape}')\n return out\n\n\ndef parse_vectors_from_directory_memory_friendly(embeddings_dir, size=None):\n paths = get_vectors_file_paths_in_vector_directory(embeddings_dir)\n if size is None:\n size = 0\n for j, p in enumerate(paths):\n logger.info(f'Loading vectors from {p} ({j+1} / {len(paths)}) to find total num vectors')\n m = torch.load(p)\n size += m.shape[0]\n\n out = None\n offset = 0\n for j, p in enumerate(paths):\n logger.info(f'Loading vectors from {p} ({j+1} / {len(paths)})')\n m = torch.load(p)\n\n assert int(p.split('.')[-1]) == j, (p, j)\n if out is None:\n out = torch.zeros(size, m.shape[1])\n out[offset: offset + m.shape[0]] = m\n offset += m.shape[0]\n assert offset == size\n logger.info(f'loaded index of shape {out.shape}')\n\n return out\n\n\ndef parse_vectors_from_directory(fi, memory_friendly=False, size=None, as_chunks=False, half=False):\n assert os.path.isdir(fi), f\"Vectors directory {fi} doesnt exist, or is not a directory of pytorch vectors\"\n if as_chunks:\n return parse_vectors_from_directory_chunks(fi, half)\n\n if memory_friendly:\n out = parse_vectors_from_directory_memory_friendly(fi, size=size)\n else:\n out = parse_vectors_from_directory_fast(fi)\n\n if half:\n out = out if out.dtype == torch.float16 else out.half()\n else:\n out = out if out.dtype == torch.float32 else out.float()\n\n return out\n\n\ndef get_submitit_executor(n_jobs=10, comment=\"\", partition='learnfair'):\n if not is_submitit_available():\n raise Exception('Submitit Not installed')\n executor = submitit.AutoExecutor(folder='PAQ_embedding_jobs')\n executor.update_parameters(timeout_min=120,\n slurm_partition=partition,\n slurm_nodes=1,\n slurm_ntasks_per_node=1,\n slurm_cpus_per_task=10,\n slurm_constraint='volta32gb',\n slurm_gpus_per_node='volta:1',\n slurm_array_parallelism=n_jobs,\n slurm_comment=comment,\n slurm_mem='64G')\n return executor\n\n" }, { "path": "paq/rerankers/__init__.py", "content": "#!/usr/bin/env python3\n# Copyright (c) Facebook, Inc. and its affiliates.\n# All rights reserved.\n#\n# This source code is licensed under the license found in the\n# LICENSE file in the root directory of this source tree." }, { "path": "paq/rerankers/rerank.py", "content": "#!/usr/bin/env python3\n# Copyright (c) Facebook, Inc. and its affiliates.\n# All rights reserved.\n#\n# This source code is licensed under the license found in the\n# LICENSE file in the root directory of this source tree.\nimport argparse\nimport torch\nimport logging\nimport os\nimport time\nfrom paq.paq_utils import is_apex_available, load_jsonl, dump_jsonl, get_submitit_executor, to_fp16\nfrom transformers import AutoConfig, AutoTokenizer, AutoModelForMultipleChoice\nif is_apex_available():\n import apex\n from apex import amp\n apex.amp.register_half_function(torch, \"einsum\")\n\nlogger = logging.getLogger(__name__)\nCUDA = torch.cuda.is_available()\n\n\ndef load_reranker(model_name_or_path):\n logger.info(f'Loading model from: {model_name_or_path}')\n config = AutoConfig.from_pretrained(model_name_or_path)\n tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, do_lower_case=True)\n model = AutoModelForMultipleChoice.from_pretrained(\n model_name_or_path,\n from_tf=bool(\".ckpt\" in model_name_or_path),\n config=config,\n )\n model = model.eval()\n return model, tokenizer\n\n\ndef get_output_format(qas, prediction_indices, prediction_scores):\n assert len(qas) == len(prediction_indices)\n return [\n {\n 'question': q['input_qa']['question'],\n 'prediction': q['retrieved_qas'][p]['answer'][0],\n 'score':s, 'index': int(p)\n }\n for q, p, s in zip(qas, prediction_indices, prediction_scores)\n ]\n\n\ndef tokenize(tokenizer, batch_qas, cuda, top_k):\n input_as, input_bs = [], []\n\n for item in batch_qas:\n question_a = item['input_qa']['question'] + '?'\n question_bs = [q['question'] + '? ' + q['answer'][0] for q in item['retrieved_qas']]\n question_bs = question_bs[:top_k]\n input_as += [question_a for _ in range(len(question_bs))]\n input_bs += question_bs\n\n inputs = tokenizer.batch_encode_plus(\n list(zip(input_as, input_bs)), return_tensors='pt', padding='longest', add_special_tokens=True\n )\n inputs = {k: v.reshape(len(batch_qas), v.shape[0]//len(batch_qas), -1) for k,v in inputs.items()}\n return {k: v.cuda() for k, v in inputs.items()} if cuda else inputs\n\n\ndef predict(model, tokenizer, qas, cuda=CUDA, bsz=16, fp16=False, top_k=30):\n\n if cuda:\n model = model.cuda()\n model = to_fp16(model) if fp16 else model\n\n t = time.time()\n\n def log_progress(j, outputs):\n t2 = time.time()\n logger.info(\n f'Reranked {j + 1} / {len(list(range(0, len(qas), bsz)))} batches in {t2 - t:0.2f} seconds '\n f'({len(outputs) / (t2 - t): 0.4f} QAs per second)')\n\n def forward(inputs):\n logits = model(**inputs)[0]\n scores, inds = logits.topk(1, dim=1)\n scores, inds = scores.squeeze().tolist(), inds.squeeze().tolist()\n if padded_batch:\n scores, inds = scores[:-1], inds[:-1]\n return scores, inds\n\n outputs = []\n output_scores = []\n logger.info(f'Embedding {len(qas)} inputs in {len(list(range(0, len(qas), bsz)))} batches:')\n with torch.no_grad():\n for j, batch_start in enumerate(range(0, len(qas), bsz)):\n\n batch = qas[batch_start: batch_start + bsz]\n padded_batch = len(batch) == 1\n if padded_batch: # hack for batch size 1 issues\n batch = [batch[0],batch[0]]\n\n inputs = tokenize(tokenizer, batch, cuda, top_k)\n scores, inds = forward(inputs)\n\n outputs.extend(inds)\n output_scores.extend(scores)\n\n log_progress(j, outputs) if j % 1 == 0 else None\n\n log_progress(j, outputs)\n\n return get_output_format(qas, outputs, output_scores)\n\n\ndef run_predictions(qas_to_rerank_file, output_file, model_name_or_path, batch_size, fp16, top_k):\n qas_to_rerank = load_jsonl(qas_to_rerank_file)\n reranker_model, reranker_tokenizer = load_reranker(model_name_or_path)\n\n predictions = predict(\n reranker_model,\n reranker_tokenizer,\n qas_to_rerank,\n bsz=batch_size,\n fp16=fp16,\n top_k=top_k\n )\n dump_jsonl(predictions, output_file)\n\n\ndef parse_files(args):\n infis, outfis = args.qas_to_rerank.split(','), args.output_files.split(',')\n assert len(infis) == len(outfis)\n pairs = []\n for in_fi, out_fi in zip(infis, outfis):\n if os.path.exists(out_fi):\n logging.info(f'skipping inference on {out_fi}, file exists')\n pairs.append((in_fi, out_fi))\n return pairs\n\n\nif __name__ == '__main__':\n parser = argparse.ArgumentParser(\"Perform RePAQ Reranking. This program will rerank retrieval results from retrieve.py.\")\n parser.add_argument('--model_name_or_path', type=str,)\n parser.add_argument('--qas_to_rerank', type=str, help='comma separated list of files produced by retrieve.py to rerank')\n parser.add_argument('--output_files', type=str, help='comma separated list of filenames to write, one for each filenmae in --qas_to_rerank')\n parser.add_argument('--top_k', type=int, default=50, help='top k to rerank')\n parser.add_argument('--fp16', action='store_true')\n parser.add_argument('--batch_size', type=int, default=8)\n parser.add_argument('--n_jobs', type=int, required=True, help='how many parallel jobs to use in slurm (n_jobs=-1 will run locally)')\n parser.add_argument('--slurm_partition', type=str, default=\"learnfair\", help='If using submitit to run slurm jobs, define cluster partition here')\n parser.add_argument('--slurm_comment', type=str, default=\"\", help='If using submitit to run slurm jobs, define job comment here')\n parser.add_argument('-v', '--verbose', action=\"store_true\")\n args = parser.parse_args()\n\n if args.verbose:\n logging.basicConfig(level=logging.DEBUG)\n\n pairs = parse_files(args)\n\n if args.n_jobs != -1:\n executor = get_submitit_executor(n_jobs=args.n_jobs, comment=args.slurm_comment, partition=args.slurm_partition)\n with executor.batch():\n jobs = [\n executor.submit(run_predictions, infi, outfi, args.model_name_or_path, args.batch_size, args.fp16, args.top_k)\n for infi, outfi in pairs\n ]\n logger.info('launched the following jobs:')\n [logger.info(job.job_id) for job in jobs]\n else:\n for infi, outfi in pairs:\n run_predictions(infi, outfi, args.model_name_or_path, args.batch_size, args.fp16, args.top_k)\n" }, { "path": "paq/retrievers/__init__.py", "content": "#!/usr/bin/env python3\n# Copyright (c) Facebook, Inc. and its affiliates.\n# All rights reserved.\n#\n# This source code is licensed under the license found in the\n# LICENSE file in the root directory of this source tree." }, { "path": "paq/retrievers/build_index.py", "content": "#!/usr/bin/env python3\n# Copyright (c) Facebook, Inc. and its affiliates.\n# All rights reserved.\n#\n# This source code is licensed under the license found in the\n# LICENSE file in the root directory of this source tree.\nimport argparse\nimport torch\nimport logging\nimport faiss\nimport os\nimport random\nfrom paq.paq_utils import parse_vectors_from_directory\n\nlogging.basicConfig(level=logging.DEBUG)\nlogger = logging.getLogger(__name__)\n\n\ndef get_vector_sample(cached_embeddings_path, sample_fraction):\n samples = []\n max_phi = -1\n N = 0\n vectors = parse_vectors_from_directory(cached_embeddings_path, as_chunks=True)\n for chunk in vectors:\n phis = (chunk ** 2).sum(1)\n max_phi = max(max_phi, phis.max())\n N += chunk.shape[0]\n if sample_fraction == 1.0:\n chunk_sample = chunk\n else:\n chunk_sample = chunk[random.sample(range(0, len(chunk)), int(len(chunk) * sample_fraction))]\n samples.append(chunk_sample)\n\n del vectors\n vector_sample = torch.cat(samples)\n return vector_sample, max_phi, N\n\n\ndef get_vectors_dim(cached_embeddings_path):\n vectors = parse_vectors_from_directory(cached_embeddings_path, as_chunks=True)\n vector_size = next(vectors).shape[1]\n del(vectors)\n return vector_size\n\n\ndef augment_vectors(vectors, max_phi):\n phis = (vectors ** 2).sum(1)\n aux_dim = torch.sqrt(max_phi - phis)\n vectors = torch.cat([vectors, aux_dim.unsqueeze(-1)], -1)\n return vectors\n\n\ndef build_index_streaming(cached_embeddings_path,\n output_path,\n hnsw=False,\n sq8_quantization=False,\n fp16_quantization=False,\n store_n=256,\n ef_search=32,\n ef_construction=80,\n sample_fraction=0.1,\n indexing_batch_size=5000000,\n ):\n\n vector_size = get_vectors_dim(cached_embeddings_path)\n\n if hnsw:\n if sq8_quantization:\n index = faiss.IndexHNSWSQ(vector_size + 1, faiss.ScalarQuantizer.QT_8bit, store_n)\n elif fp16_quantization:\n index = faiss.IndexHNSWSQ(vector_size + 1, faiss.ScalarQuantizer.QT_fp16, store_n)\n else:\n index = faiss.IndexHNSWFlat(vector_size + 1, store_n)\n\n index.hnsw.efSearch = ef_search\n index.hnsw.efConstruction = ef_construction\n else:\n if sq8_quantization:\n index = faiss.IndexScalarQuantizer(vector_size, faiss.ScalarQuantizer.QT_8bit, faiss.METRIC_L2)\n elif fp16_quantization:\n index = faiss.IndexScalarQuantizer(vector_size, faiss.ScalarQuantizer.QT_fp16, faiss.METRIC_L2)\n else:\n index = faiss.IndexIP(vector_size + 1, store_n)\n\n vector_sample, max_phi, N = get_vector_sample(cached_embeddings_path, sample_fraction)\n if hnsw:\n vector_sample = augment_vectors(vector_sample, max_phi)\n\n if sq8_quantization or fp16_quantization: # index requires training\n vs = vector_sample.numpy()\n logging.info(f'Training Quantizer with matrix of shape {vs.shape}')\n index.train(vs)\n del vs\n del vector_sample\n\n chunks_to_add = []\n added = 0\n for vector_chunk in parse_vectors_from_directory(cached_embeddings_path, as_chunks=True):\n if hnsw:\n vector_chunk = augment_vectors(vector_chunk, max_phi)\n\n chunks_to_add.append(vector_chunk)\n\n if sum(c.shape[0] for c in chunks_to_add) > indexing_batch_size:\n logging.info(f'Adding Vectors {added} -> {added + to_add.shape[0]} of {N}')\n to_add = torch.cat(chunks_to_add)\n chunks_to_add = []\n index.add(to_add)\n added += 1\n\n if len(chunks_to_add) > 0:\n to_add = torch.cat(chunks_to_add).numpy()\n index.add(to_add)\n logging.info(f'Adding Vectors {added} -> {added + to_add.shape[0]} of {N}')\n\n logger.info(f'Index Built, writing index to {output_path}')\n faiss.write_index(index, output_path)\n logger.info(f'Index dumped')\n return index\n\n\nif __name__ == '__main__':\n parser = argparse.ArgumentParser(\"Build a FAISS index from precomputed vector files from embed.py. \"\n \"Provides functionality to build either flat indexes (slow but exact)\"\n \" or HNSW indexes (much faster, but approximate). \"\n \"Optional application of 8bit or 16bit quantization is also available.\"\n \" Many more indexes are possible with Faiss, consult the Faiss repository here\"\n \" if you want to build more advanced indexes.\")\n parser.add_argument('--embeddings_dir', type=str, help='path to directory containing vectors to build index from')\n parser.add_argument('--output_path', type=str, help='path to write results to')\n parser.add_argument('--hnsw', action='store_true', help='Build an HNSW index rather than Flat')\n parser.add_argument('--SQ8', action='store_true', help='use SQ8 quantization on index to save memory')\n parser.add_argument('--fp16', action='store_true', help='use fp16 quantization on index to save memory')\n parser.add_argument('--store_n', type=int, default=32, help='hnsw store_n parameter')\n parser.add_argument('--ef_construction', type=int, default=128, help='hnsw ef_construction parameter')\n parser.add_argument('--ef_search', type=int, default=128, help='hnsw ef_search parameter')\n parser.add_argument('--sample_fraction', type=float, default=1.0,\n help='If memory is limited, specify a fraction (0.0->1.0) of the '\n 'data to sample for training the quantizer')\n parser.add_argument('--indexing_batch_size', type=int, default=None,\n help='If memory is limited, specify the approximate number '\n 'of vectors to add to the index at once')\n parser.add_argument('-v', '--verbose', action=\"store_true\")\n args = parser.parse_args()\n\n if args.verbose:\n logging.basicConfig(level=logging.DEBUG)\n assert not (args.SQ8 and args.fp16), 'cant use both sq8 and fp16 Quantization'\n assert not os.path.exists(args.output_path), \"Faiss index with name specificed in --output_path already exists\"\n\n args.indexing_batch_size = 10000000000000 if args.indexing_batch_size is None else args.indexing_batch_size\n assert 0 < args.sample_fraction <= 1.0\n\n if args.sample_fraction:\n build_index_streaming(\n args.embeddings_dir,\n args.output_path,\n args.hnsw,\n sq8_quantization=args.SQ8,\n fp16_quantization=args.fp16,\n store_n=args.store_n,\n ef_construction=args.ef_construction,\n ef_search=args.ef_search,\n sample_fraction=args.sample_fraction,\n indexing_batch_size=args.indexing_batch_size,\n )\n" }, { "path": "paq/retrievers/embed.py", "content": "#!/usr/bin/env python3\n# Copyright (c) Facebook, Inc. and its affiliates.\n# All rights reserved.\n#\n# This source code is licensed under the license found in the\n# LICENSE file in the root directory of this source tree.\nimport argparse\nimport torch\nimport logging\nimport time\nimport math\nimport os\nfrom paq.paq_utils import is_apex_available, load_jsonl, get_submitit_executor, to_fp16\nfrom paq.retrievers.retriever_utils import load_retriever\n\n\nlogger = logging.getLogger(__name__)\nCUDA = torch.cuda.is_available()\n\n\ndef embed(model, tokenizer, qas, bsz=256, cuda=CUDA, fp16=False):\n\n def normalize_q(question: str) -> str:\n return question.strip().strip('?').lower().strip()\n\n def tokenize(batch_qas):\n input_qs = [normalize_q(q['question']) for q in batch_qas]\n inputs = tokenizer.batch_encode_plus(\n input_qs, return_tensors='pt', padding=True, add_special_tokens=True\n )\n return {k: v.cuda() for k, v in inputs.items()} if cuda else inputs\n\n if cuda:\n model = model.cuda()\n model = to_fp16(model) if fp16 else model\n\n t = time.time()\n\n def log_progress(j, outputs):\n t2 = time.time()\n logger.info(\n f'Embedded {j + 1} / {len(list(range(0, len(qas), bsz)))} batches in {t2 - t:0.2f} seconds '\n f'({sum([len(o) for o in outputs]) / (t2 - t): 0.4f} QAs per second)')\n\n outputs = []\n with torch.no_grad():\n for j, batch_start in enumerate(range(0, len(qas), bsz)):\n batch_qas = qas[batch_start: batch_start + bsz]\n inputs = tokenize(batch_qas)\n batch_outputs = model(**inputs)\n outputs.append(batch_outputs.cpu())\n if j % 10 == 0:\n log_progress(j, outputs)\n\n log_progress(j, outputs)\n\n return torch.cat(outputs, dim=0).cpu()\n\n\ndef embed_job(qas_to_embed_path, model_name_or_path, output_file_name, n_jobs, job_num, batch_size, fp16, memory_friendly_parsing):\n os.makedirs(os.path.dirname(output_file_name), exist_ok=True)\n\n qas_to_embed = load_jsonl(qas_to_embed_path, memory_friendly=memory_friendly_parsing)\n chunk_size = math.ceil(len(qas_to_embed) / n_jobs)\n\n qas_to_embed_this_job = qas_to_embed[job_num * chunk_size: (job_num + 1) * chunk_size]\n logger.info(f'Embedding Job {job_num}: Embedding {len(qas_to_embed)} inputs in {int(len(qas_to_embed) / batch_size)} batches:')\n\n model, tokenizer = load_retriever(model_name_or_path)\n mat = embed(model, tokenizer, qas_to_embed_this_job, bsz=batch_size, fp16=fp16)\n torch.save(mat.half(), output_file_name + f'.{job_num}')\n\n\nif __name__ == '__main__':\n parser = argparse.ArgumentParser()\n parser.add_argument('--model_name_or_path', type=str, required=True, help='path to HF model dir')\n parser.add_argument('--qas_to_embed', type=str,required=True, help='Path to questions to embed in jsonl format')\n parser.add_argument('--n_jobs', type=int, required=True, help='how many jobs to embed with (n_jobs=-1 will run a single job locally)')\n parser.add_argument('--output_dir', type=str, help='path to write vectors to')\n parser.add_argument('--fp16', action='store_true')\n parser.add_argument('--batch_size', type=int, default=128)\n parser.add_argument('--memory_friendly_parsing', action='store_true', help='Pass this to load jsonl files more slowly, but save memory')\n parser.add_argument('--slurm_partition', type=str, default=\"learnfair\", help='If using submitit to run slurm jobs, define cluster partition here')\n parser.add_argument('--slurm_comment', type=str, default=\"\", help='If using submitit to run slurm jobs, define job comment heree')\n parser.add_argument('-v', '--verbose', action=\"store_true\")\n args = parser.parse_args()\n\n if args.verbose:\n logging.basicConfig(level=logging.DEBUG)\n\n if args.fp16 and not CUDA:\n raise Exception('Cant use --fp16 without a gpu, CUDA not found')\n\n output_path = os.path.join(args.output_dir, 'embeddings.pt')\n\n if args.n_jobs == -1:\n embed_job(\n args.qas_to_embed,\n args.model_name_or_path,\n output_path,\n n_jobs=1,\n job_num=0,\n batch_size=args.batch_size,\n fp16=args.fp16,\n memory_friendly_parsing=args.memory_friendly_parsing\n )\n else:\n executor = get_submitit_executor(n_jobs=args.n_jobs, comment=args.slurm_comment, partition=args.slurm_partition)\n jobs = []\n with executor.batch():\n for jn in range(args.n_jobs):\n job = executor.submit(\n embed_job,\n args.qas_to_embed,\n args.model_name_or_path,\n output_path,\n args.n_jobs,\n jn,\n args.batch_size,\n args.fp16,\n args.memory_friendly_parsing\n )\n jobs.append(job)\n\n logger.info('launched the following jobs:')\n for job in jobs:\n logger.info(job.job_id)\n" }, { "path": "paq/retrievers/retrieve.py", "content": "#!/usr/bin/env python3\n# Copyright (c) Facebook, Inc. and its affiliates.\n# All rights reserved.\n#\n# This source code is licensed under the license found in the\n# LICENSE file in the root directory of this source tree.\nimport argparse\nimport torch\nimport logging\nimport time\nimport faiss\nimport numpy as np\nfrom paq.retrievers.retriever_utils import load_retriever\nfrom paq.paq_utils import load_jsonl, dump_jsonl, parse_vectors_from_directory\nfrom paq.retrievers.embed import embed\nfrom copy import deepcopy\n\nlogger = logging.getLogger(__name__)\n\nCUDA = torch.cuda.is_available()\n\n\ndef get_output_format(qas_to_answer, qas_to_retrieve_from, top_indices, top_scores):\n results = []\n for qa_ind, qa in enumerate(qas_to_answer):\n res = []\n for score_ind, ind in enumerate(top_indices[qa_ind]):\n score = top_scores[qa_ind][score_ind]\n ret_qa = deepcopy(qas_to_retrieve_from[ind])\n ret_qa['score'] = float(score)\n res.append(ret_qa)\n results.append(res)\n\n return [{'input_qa': in_qa, 'retrieved_qas': ret_qas} for in_qa, ret_qas in zip(qas_to_answer, results)]\n\n\ndef _torch_mips(index, query_batch, top_k):\n sims = torch.matmul(query_batch, index.t())\n return sims.topk(top_k)\n\n\ndef _flat_index_mips(index, query_batch, top_k):\n return index.search(query_batch.numpy(), top_k)\n\n\ndef _aux_dim_index_mips(index, query_batch, top_k):\n # querying faiss indexes for MIPS using a euclidean distance index, used with hnsw\n aux_dim = query_batch.new(query_batch.shape[0]).fill_(0)\n aux_query_batch = torch.cat([query_batch, aux_dim.unsqueeze(-1)], -1)\n return index.search(aux_query_batch.numpy(), top_k)\n\n\ndef _get_mips_function(index):\n if type(index) == torch.Tensor:\n return _torch_mips\n elif 'hnsw' in str(type(index)).lower():\n return _aux_dim_index_mips\n else:\n return _flat_index_mips\n\n\ndef mips(index, queries, top_k, n_queries_to_parallelize=256):\n t = time.time()\n all_top_indices = None\n all_top_scores = None\n\n _mips = _get_mips_function(index)\n\n for mb in range(0, len(queries), n_queries_to_parallelize):\n query_batch = queries[mb:mb + n_queries_to_parallelize].float()\n scores, top_indices = _mips(index, query_batch, top_k)\n\n all_top_indices = top_indices if all_top_indices is None else np.concatenate([all_top_indices, top_indices])\n all_top_scores = scores if all_top_scores is None else np.concatenate([all_top_scores, scores])\n\n delta = time.time() - t\n logger.info(\n f'{len(all_top_indices)}/ {len(queries)} queries searched in {delta:04f} '\n f'seconds ({len(all_top_indices) / delta} per second)')\n\n assert len(all_top_indices) == len(queries)\n\n delta = time.time() - t\n logger.info(f'Index searched in {delta:04f} seconds ({len(queries) / delta} per second)')\n return all_top_indices, all_top_scores\n\n\ndef run_queries(model, tokenizer, qas_to_retrieve_from, qas_to_answer, top_k, index=None,\n batch_size=128, fp16=False, n_queries_to_parallelize=2048):\n\n if index is None:\n index = embed(model, tokenizer, qas_to_retrieve_from, bsz=batch_size, fp16=fp16).float()\n\n logger.info('Embedding QAs to answer:')\n embedded_qas_to_answer = embed(model, tokenizer, qas_to_answer, bsz=batch_size, fp16=fp16)\n logger.info('Running MIPS search:')\n top_indices, top_scores = mips(index, embedded_qas_to_answer, top_k, n_queries_to_parallelize=n_queries_to_parallelize)\n\n return get_output_format(qas_to_answer, qas_to_retrieve_from, top_indices, top_scores)\n\n\ndef _load_index_if_exists(faiss_index_path, precomputed_embeddings_dir, n_vectors_to_load=None, memory_friendly=False, efsearch=128):\n index = None\n if faiss_index_path is not None:\n assert precomputed_embeddings_dir is None, \"Do not specify both a --faiss_index_path and --precomputed_embeddings_dir\"\n logger.info('Loading Faiss index:')\n index = faiss.read_index(faiss_index_path)\n if hasattr(index, 'hnsw'):\n index.hnsw.efSearch = efsearch\n\n elif precomputed_embeddings_dir is not None:\n logger.info('Loading vectors index from file:')\n index = parse_vectors_from_directory(\n precomputed_embeddings_dir,\n memory_friendly=memory_friendly,\n size=n_vectors_to_load\n ).float()\n\n logger.info('Index loaded') if index is not None else None\n return index\n\n\nif __name__ == '__main__':\n parser = argparse.ArgumentParser(\n \"Perform REPAQ QA-Pair Retrieval. This program will embed a file of questions which need\"\n \" answering passed as `--qas_to_answer`. These will be answered by retrieving QA-pairs from a \"\n \" set of QA pairs to retrieve answers from, passed in as `--qas_to_retrieve_from`. \"\n \" The program can retrieve either from a prebuilt faiss index for `qas_to_retrieve_from`, \"\n \"or a directory of precomputed vectors, or, if neither are passed in, \"\n \"will embed the `qas_to_retrieve_from` before performing retrieval\"\n )\n parser.add_argument('--model_name_or_path', type=str, required=True, help='path to HF model dir')\n parser.add_argument('--qas_to_answer', type=str, required=True, help=\"path to questions to answer in jsonl format\")\n parser.add_argument('--qas_to_retrieve_from', type=str, required=True,\n help=\"path to QA-pairs to retrieve answers from in jsonl format\")\n parser.add_argument('--top_k', type=int, default=50, help=\"top K QA-pairs to retrieve for each input question\")\n parser.add_argument('--output_file', type=str, required=True, help='Path to write jsonl results to')\n parser.add_argument('--faiss_index_path', default=None, type=str, help=\"Path to faiss index, if retrieving from a faiss index\")\n parser.add_argument('--precomputed_embeddings_dir', default=None, type=str, help=\"path to a directory of vector embeddings if retrieving from raw embeddign vectors\")\n parser.add_argument('--fp16', action='store_true')\n parser.add_argument('--batch_size', type=int, default=128, help='Batch size for embedding questions for querying')\n parser.add_argument('--n_queries_to_parallelize', type=int, default=256, help=\"query batch size\")\n parser.add_argument('-v', '--verbose', action=\"store_true\")\n parser.add_argument('--memory_friendly_parsing', action='store_true', help='Pass this to load files more slowly, but save memory')\n parser.add_argument('--faiss_efsearch', type=int, default=128, help='EFSearch searchtime parameter for hnsw , higher is more accuate but slower')\n\n args = parser.parse_args()\n\n if args.verbose:\n logging.basicConfig(level=logging.DEBUG)\n\n qas_to_answer = load_jsonl(args.qas_to_answer, memory_friendly=args.memory_friendly_parsing)\n qas_to_retrieve_from = load_jsonl(args.qas_to_retrieve_from, memory_friendly=args.memory_friendly_parsing)\n\n index = _load_index_if_exists(\n args.faiss_index_path,\n args.precomputed_embeddings_dir,\n n_vectors_to_load=len(qas_to_retrieve_from),\n memory_friendly=args.memory_friendly_parsing,\n efsearch=args.faiss_efsearch\n )\n model, tokenizer = load_retriever(args.model_name_or_path)\n\n retrieved_answers = run_queries(\n model,\n tokenizer,\n qas_to_retrieve_from,\n qas_to_answer,\n args.top_k,\n index,\n args.batch_size,\n args.fp16,\n args.n_queries_to_parallelize,\n )\n dump_jsonl(retrieved_answers, args.output_file)\n" }, { "path": "paq/retrievers/retriever_utils.py", "content": "#!/usr/bin/env python3\n# Copyright (c) Facebook, Inc. and its affiliates.\n# All rights reserved.\n#\n# This source code is licensed under the license found in the\n# LICENSE file in the root directory of this source tree.\nimport torch\nfrom torch import nn\nimport os\nimport logging\nfrom transformers import AutoConfig, AutoTokenizer, AutoModel\n\n\nlogger = logging.getLogger(__name__)\n\n\ndef _get_proj_keys_from_state_dict(state_dict):\n weight_key = [k for k in state_dict.keys() if 'encode_proj' in k and 'weight' in k]\n bias_key = [k for k in state_dict.keys() if 'encode_proj' in k and 'bias' in k]\n assert len(weight_key) == 1 == len(bias_key)\n weight_key, bias_key = weight_key[0], bias_key[0]\n return weight_key, bias_key\n\n\ndef _get_proj_dim_from_model_path(model_name_or_path):\n state = torch.load(os.path.join(model_name_or_path, 'pytorch_model.bin'), map_location=torch.device('cpu'))\n proj_dim = None\n if any('encode_proj' in k for k in state.keys()):\n _, bias_key = _get_proj_keys_from_state_dict(state)\n proj_dim = state[bias_key].shape[0]\n return proj_dim\n\n\ndef load_retriever(model_name_or_path):\n logger.info(f'Loading model from: {model_name_or_path}')\n model = RetrieverEncoder.from_pretrained(model_name_or_path)\n tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, do_lower_case=True)\n model.eval()\n return model, tokenizer\n\n\nclass RetrieverEncoder(nn.Module):\n \"\"\"A wrapper for HF models, with an optional projection\"\"\"\n\n def __init__(self, config, proj_dim):\n super().__init__()\n # EncoderBase.__init__(self, config.hidden_size, project_dim)\n self.model = AutoModel.from_config(config)\n self.encode_proj = nn.Linear(config.hidden_size, proj_dim) if proj_dim is not None else None\n self.model.init_weights()\n\n @classmethod\n def from_pretrained(cls, model_name_or_path):\n config = AutoConfig.from_pretrained(model_name_or_path)\n proj_dim = _get_proj_dim_from_model_path(model_name_or_path)\n\n retriever = cls(config, proj_dim)\n state = torch.load(os.path.join(model_name_or_path, 'pytorch_model.bin'), map_location=torch.device('cpu'))\n retriever.model.load_state_dict({k.replace('albert.',''):v for k,v in state.items() if 'encode_proj' not in k}, strict=True)\n\n if proj_dim is not None:\n weight_key, bias_key = _get_proj_keys_from_state_dict(state)\n retriever.encode_proj.load_state_dict({'weight': state[weight_key], 'bias': state[bias_key]}, strict=True)\n\n return retriever\n\n def forward(self, *args, **kwargs):\n seq_outputs = self.model(*args, **kwargs)['last_hidden_state']\n return self.encode_proj(seq_outputs[:, 0]) if self.encode_proj is not None else seq_outputs[:, 0]\n" }, { "path": "paq/server/__init__.py", "content": "#!/usr/bin/env python3\n# Copyright (c) Facebook, Inc. and its affiliates.\n# All rights reserved.\n#\n# This source code is licensed under the license found in the\n# LICENSE file in the root directory of this source tree." }, { "path": "paq/server/client.py", "content": "#!/usr/bin/env python3\n# Copyright (c) Facebook, Inc. and its affiliates.\n# All rights reserved.\n#\n# This source code is licensed under the license found in the\n# LICENSE file in the root directory of this source tree.\nimport requests\nimport time\n\ntask = {\"query\": \"who sang does he love me with reba\", \"k\": 10}\nresp = requests.post(\"http://0.0.0.0:1359/\", json=task)\ndata = resp.json()\nprint(data)\n" }, { "path": "paq/server/launch_server.sh", "content": "#!/usr/bin/env bash\n# Copyright (c) Facebook, Inc. and its affiliates.\n# All rights reserved.\n#\n# This source code is licensed under the license found in the\n# LICENSE file in the root directory of this source tree.\npython -m paq.download -v -n models.retrievers.retriever_multi_base_256\npython -m paq.download -v -n paq.TQA_TRAIN_NQ_TRAIN_PAQ\npython -m paq.download -v -n indices.multi_base_256_hnsw_sq8\n\npython -m paq.server.server \\\n --model_name_or_path data/models/retrievers/retriever_multi_base_256 \\\n --qas_to_retrieve_from data/paq/TQA_TRAIN_NQ_TRAIN_PAQ/tqa-train-nq-train-PAQ.jsonl \\\n --top_k 10 \\\n --faiss_index_path data/indices/multi_base_256_hnsw_sq8.faiss \\\n --fp16 \\\n --memory_friendly_parsing \\\n --verbose" }, { "path": "paq/server/server.py", "content": "#!/usr/bin/env python3\n# Copyright (c) Facebook, Inc. and its affiliates.\n# All rights reserved.\n#\n# This source code is licensed under the license found in the\n# LICENSE file in the root directory of this source tree.\nfrom http.server import BaseHTTPRequestHandler, HTTPServer, HTTPStatus\nimport json\nimport argparse\nfrom paq.retrievers.retrieve import _load_index_if_exists, load_retriever, load_jsonl, run_queries\nimport logging\n\nlogger = logging.getLogger(__name__)\n\n\nclass http_server:\n def __init__(self, index, model, tokenizer, qas_to_retrieve_from, fp16):\n server = HTTPServer((\"\", 1359), WebServerHandler)\n server.index = index\n server.model = model\n server.tokenizer = tokenizer\n server.qas_to_retrieve_from = qas_to_retrieve_from\n server.fp16 = fp16\n logging.info(\"Web Server running:\")\n server.serve_forever()\n\n\nclass WebServerHandler(BaseHTTPRequestHandler):\n\n # POST echoes the message adding a JSON field\n def do_POST(self):\n datalen = int(self.headers[\"Content-Length\"])\n data = self.rfile.read(datalen)\n obj = json.loads(data)\n logger.info(\"Got object: {}\".format(obj))\n\n if \"query\" in obj and \"k\" in obj:\n qas_to_answer = [{'question': obj['query']}]\n result = run_queries(\n self.server.model,\n self.server.tokenizer,\n self.server.qas_to_retrieve_from,\n qas_to_answer,\n top_k=obj['k'],\n index=self.server.index,\n batch_size=1,\n fp16=args.fp16,\n n_queries_to_parallelize=1\n )\n logger.info(\"result: \" + json.dumps(result))\n\n # send the message back\n self.send_response(200)\n self.end_headers()\n self.wfile.write(json.dumps({\"result\": result}).encode())\n return\n else:\n self.send_response(HTTPStatus.BAD_REQUEST)\n self.end_headers()\n\n\ndef main(args):\n qas_to_retrieve_from = load_jsonl(args.qas_to_retrieve_from, memory_friendly=args.memory_friendly_parsing)\n index = _load_index_if_exists(\n args.faiss_index_path,\n args.precomputed_embeddings_dir,\n n_vectors_to_load=len(qas_to_retrieve_from),\n memory_friendly=args.memory_friendly_parsing,\n efsearch=args.faiss_efsearch\n )\n model, tokenizer = load_retriever(args.model_name_or_path)\n\n try:\n server = http_server(index, model, tokenizer, qas_to_retrieve_from, args.fp16)\n logging.info(\"Web Server running:\")\n except KeyboardInterrupt:\n logging.info(\" ^C entered, stopping web server....\")\n server.server.socket.close()\n\n\nif __name__ == \"__main__\":\n parser = argparse.ArgumentParser(\"Server that wraps Retrieval functionality\")\n parser.add_argument('--model_name_or_path', type=str, required=True, help='path to HF model dir')\n parser.add_argument('--qas_to_retrieve_from', type=str, required=True,\n help=\"path to QA-pairs to retrieve answers from in jsonl format\")\n parser.add_argument('--top_k', type=int, default=50, help=\"top K QA-pairs to retrieve for each input question\")\n parser.add_argument('--faiss_index_path', default=None, type=str,\n help=\"Path to faiss index, if retrieving from a faiss index\")\n parser.add_argument('--precomputed_embeddings_dir', default=None, type=str,\n help=\"path to a directory of vector embeddings if retrieving from raw embeddign vectors\")\n parser.add_argument('--fp16', action='store_true')\n parser.add_argument('--batch_size', type=int, default=128, help='Batch size for embedding questions for querying')\n parser.add_argument('--n_queries_to_parallelize', type=int, default=256, help=\"query batch size\")\n parser.add_argument('-v', '--verbose', action=\"store_true\")\n parser.add_argument('--memory_friendly_parsing', action='store_true',\n help='Pass this to load files more slowly, but save memory')\n parser.add_argument('--faiss_efsearch', type=int, default=128,\n help='EFSearch searchtime parameter for hnsw , higher is more accuate but slower')\n args = parser.parse_args()\n if args.verbose:\n logging.basicConfig(level=logging.DEBUG)\n\n main(args)\n" }, { "path": "requirements.txt", "content": "wget>=3.2\ntransformers==4.1.0\nsentencepiece\nprotobuf\nsubmitit\nspacy" } ]